Metadata
- Author: Jack Vanlightly
- Full Title:: Data engineering is a discipline of software engineering
- Category:: 🗞️Articles
- Document Tags:: Data culture,
- URL:: https://jack-vanlightly.com/blog/2024/10/21/the-curse-of-conway-and-the-data-space
- Finished date:: 2024-12-01
Highlights
There are three key trends that have emerged over the last two years in the data analytics space that have the potential to make real improvements. Each is still quite nascent but gaining momentum: • Data engineering is a discipline of software engineering. • Data contracts and data products. • Shift Left. (View Highlight)
The idea of Shift Left is to shift the security focus left to where software is being developed, rather than being applied after the fact, by a separate team with little knowledge of the software being developed, modified, and deployed. (View Highlight)
The fact that we have to extract data from a source application and feed it into a data processing tool, a data or machine learning platform or business intelligence (BI) tools to do something meaningful with it, is actually only a workaround for inappropriate data management. A workaround necessary because of the completely inadequate way of dealing with data in the enterprise today. (View Highlight)
OMG, this!!!
Service B didn’t just reach into the private internals of Service A; instead, an interface was created that allowed Service A to get data from Service B without violating encapsulation. This interface, an API, queue, or stream, became a stable method of data consumption that didn’t break every time Service A needed to change its hidden internals. The burden of providing that interface was placed on the team of Service A because it was the right solution, but there was also a business case to do so. The same applies with Shift Left; instead of placing the ownership of making data available on the person who wants to use the data, you place that ownership upstream to where the data is produced and maintained. (View Highlight)