Metadata
- Author: Matt Arderne
- Full Title:: The future history of Data Engineering
- Category:: 🗞️Articles
- Document Tags:: Data Engineering,
- URL:: https://groupby1.substack.com/p/data-engineering
- Finished date:: 2023-08-24
Highlights
Most businesses’ data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. (View Highlight)
Most businesses’ data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. (View Highlight)
Most businesses’ data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. (View Highlight)
Most businesses’ data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. (View Highlight)
Most businesses’ data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. (View Highlight)
But most people who will build the latter will still call themselves Data Engineers
Most businesses’ data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering. (View Highlight)
Eventually, analytics engineering could face the same turn of the tide. When the tooling gets so good that the team is composed entirely of analysts and product people, and no contriving engineers. In the same way that structural engineers are only required when building on quicksand, data engineers are only required when building upon a dataswamp. As the tooling gets better, so do the foundations stabilise. (View Highlight)
The implication for engineers whose work is now easier is the following: Either you move in the direction of the new business problem. Or you move to a new business that still has the old problem. Or you specialise further until you find another domain to play in, and wait for the tide to turn again. (View Highlight)
DPE just means that you are the Tech Lead of the Analytics Engineering. (View Highlight)