- Tags:: 🗞️Articles, Data team roles, Analytics Engineering
- Author:: Claire Carroll
- Link:: What is Analytics Engineering? (getdbt.com)
- Source date:: 2019-10-16
- Finished date:: 2021-08-10
They are an hybrid of DE and DA, although that surprises me, since the qualities for analyzing are different from engineering (so I see Data Analysis closer to Data Science).
Analytics Engineering is in the middle of Data Engineer and Data Analyst (motivated by the advancements of the Modern Data Stack)
The canonical definition:
Analytics engineers provide clean data sets to end users, modeling data in a way that empowers end users to answer their own questions. While a data analyst spends their time analyzing data, an analytics engineer spends their time transforming, testing, deploying, and documenting data. Analytics engineers apply software engineering best practices like version control and continuous integration to the analytics code base.
The lines between these roles are blurry – some analytics engineers might spend time doing analyst work like deep dives, while others might be comfortable writing production level Python code but realize doing so often isn’t the highest leverage use of their time.
And the functions while be divided among the roles as follows:
But why people start thinking about analytics engineering as a specific role? “In the past”, there is no modeling, analysts owned all reporting in an ad-hoc way:
If you were on a “traditional data team” pre 2012, your first data hire was probably a data engineer. (…) Analysts, like me, would maintain a mess of SQL files with names like
monthly_revenue_final.sql, or maybe just bookmark their queries in a SQL web editor. Often we would need to supplement data in the warehouse with fancy Excel work. The people consuming the data–CEOs, Marketing VPs, CFOs–would receive monthly reports, request special analysis as-needed, and send analysts a never-ending stream of requests to “segment by this” or “cut by that” or “hey, we’ve updated our definition of ‘account’”. Being a data analyst was a hard and thankless job, and it didn’t have a ton of leverage.
Cloud warehouses, and with that, self-service puts a focus on data modeling to make data easier to analyze by others:
As data tools changed, so did the people who used them. People who weren’t on data teams began developing data literacy. This was good: business users wanted to self-serve and be data-driven. The downside was that these people often knew just enough SQL to be dangerous.
And the stack made it easier to have a somewhat full-stack data person:
Today, if you’re a “modern data team” your first data hire will be someone who ends up owning the entire data stack. This person can set up Stitch or Fivetran to start ingesting data, keep a data warehouse tidy, write complex data transformations in SQL using dbt, and build reports on top of a clean data layer in Looker, Mode, Redash, etc.
This job is neither data engineering, nor analysis. It’s somewhere in the middle, and it needed a new title. Starting in 2018, we and a few of our friends in the Locally Optimistic community started calling this role the analytics engineer.