Monday, October 3, 2022

Data engineering: why is it so important in Data Science projects?

If you follow Blog MBA USP/Esalq, then you already know some important information about Data Science, right? But do you know exactly what is the importance of data engineering for this science?

We talked to Jeronymo Marcondes, professor of the MBA in Data Science and Analytics USP/Esalq who explained why, after all, data engineering is so important for Data Science projects.

But before, if you haven’t seen, how about checking the 3 pillars of Data Science for those who want to stand out?

What is it?

After all, do you know what is data engineering? According to Marcondes: “It is the name we give to the area that works the data for the data scientist. Basically, data engineers perform the processes of extraction, transformation and data load (extract, transform, load – ETL).”

It is the data engineer who ensures that the data arrives ready to be used to data scientists.

What is it for?

And what data scientists do with the information that comes to them from data engineering? Well, Data Science is a rather broad and complex concept to summarize. But Marcondes seeks to clarify some important processes.

“In general terms, data scientists work with predictions and inferences. Based on multidisciplinary knowledge, such as mathematics, computation and statistics, the data scientist seeks to capture, store and process information based on data, and these predictions and inferences can be generated”, he explains.

The relationship of data engineering with processes

Data engineering is responsible, then, for creating the processes that generate databases from ETLs.

“The maintenance of these processes, optimization of data response and system feeding architecture are some examples of other attributes of the data engineer. Briefly, the data engineer will ensure that the data to be used by data scientists is updated, available and with efficient architecture, which facilitates their consultations”, details Marcondes.

The data engineering professional

And if you are wondering what characteristics are important to the data engineer, the professor gives somes tips!

“I believe that the main characteristics are: analytical thinking, deep knowledge of databases and methods of storage and data transfer and, mainly, the will to solve problems. The latter is the main characteristic, in my personal opinion.”

Data engineering in crises

According to Marcondes, the Covid-19 pandemic was a good example of the importance of data engineering. “Many people went to home office and needed the data to be used by a company to be available remotely”, he says.

How to ensure efficient databases for data scientists? How to ensure that the data to be used by employees and applications maintain its efficiency in consultation?

The professor responds: “Despite the indispensable role of IT infrastructure professionals (Information Technology), the role of data engineers was fundamental in these moments.”

Did you know about the work of data engineering professionals? What did you think? Leave a comment!

Author

Marina Petrocelli
Marina Petrocelli
Over 12 years have passed since my first experience with Social Communication. My first professional years were dedicated to the routines of redactions with no or little digital relevance. Plural journalism was resumed in investigating facts, write a piece and guarantee an expressive picture. The first sign of change came with the proposal of changing realities and experimenting a different way of producing. From then on, the specificities of the marketing universe became permanent. Oh! I’ve also graduated in Law School (passing the bar exam and all). But not everything is comprised tom y professional skills. As a content producer, I’m interested in good stories, from real people or series, movies and books, specially dystopias. I like to draw up travel plans and identify stars and constellations on a cell phone app. Museums, music and art in general grab my attention, as well as pop culture. O primeiro sinal de mudança veio com a proposta para mudar de realidade e experimentar um formato diferente de produzir. Daí pra frente, as particularidades do universo do marketing se tornaram permanentes. Ah! Também me formei em Direito (com inscrição na OAB e tudo). Mas nem tudo se resume às minhas habilidades profissionais. Como produtora de conteúdo, me interesso por boas histórias, de pessoas reais ou em séries, filmes e livros, especialmente distopias. Gosto de montar roteiros de viagens e reconhecer estrelas e constelações em um aplicativo no celular. Museus, música e arte no geral chamam minha atenção, assim como cultura pop.