Metadata
- Author: Bernd Wessely
- Full Title:: Data Engineering, Redefined
- Category:: 🗞️Articles
- URL:: https://towardsdatascience.com/data-engineering-redefined-643249cbbadd
- Finished date:: 2024-07-10
Highlights
I understand why but the devil is in the details. I’m not sure I agree with that.
To be very clear, data engineering should not implement business logic. (View Highlight)
What are “technical manipulations” that still are allowed for the data engineer, you might ask. I would define this as any manipulation to data that does not change or add new business information. We can still partition, bucket, reformat, normalize, index, technically aggregate, etc., but as soon as real business logic is necessary, we should address it to the application developers in the business domain responsible for the respective data set. (View Highlight)
if business logic is to be implemented in the pipelines, these systems should be considered their own applications in the enterprise. (View Highlight)