- Tags:: 📝CuratedNotes, Data Engineering
Interesting refs
- 🗞 The Cost of Cloud, a Trillion Dollar Paradox
- The Two Philosophies of Cost in Data Analytics
- Why rising cloud costs are the silent killers of data platforms | by Kris Peeters | Jul, 2022 | datamindedbe
The solution is close monitoring and good governance rules. But those things are often implemented too late.
No executive who wants to get a next promotion dares to challenge the ever-increasing expenses in data. They are labeled as old-fashioned, and they “don’t get it”. In reality, most executives don’t get it. The data emperor wears no clothes.
If you prefer to manage each low level component yourself, to keep full control and reduce cloud costs to the max, then make sure to invest in a stellar data team that can focus on building the platform and offer a great developer experience to the users of the platform.
- Standardise on something that you can easily move from platform to platform. Today, for all processing code, the humble docker container is a great abstraction layer. It’s universally supported across clouds. And you can run it yourself or go for managed solutions.
- Keep a close eye on cloud costs, and pro-actively optimise where it hurts the most. Don’t wait for the executives to do this exercise for you. Cloud vendors have tools and dashboards to do exactly this. Use them!
- Whatever you do, iterate. Don’t make 5 year plans. Don’t spend 1 year on architecture alone. Ship useful stuff. Collect feedback. Iterate. A modern data platform evolves over time.