The advent of the (Big) Data Architect
About three years ago, Maxime Beauchemin wrote the “Rise of the data engineer”. Since then the Data Engineer job has become more and more complex, domain-specific expertise has also pushed for separate job functions such as Machine Learning Engineers and the cloud has pushed the boundary of the role towards DevOps and DataOps horizon.
At the same time, the need for data from a business and product perspective has pushed the role from being reporting and analysis oriented to being production and action-oriented. Data is now more and more, leveraged in real-time to produce real-time marketing triggers, machine learning models are created to produce product recommendations, …
This increase in the demand to leverage data, and in the amount of resources being committed to that goal has created the need for a new role, meant to oversee this transformation in the use of data from insights into production. This has given rise to the Data Architect, who faced with now a plethora of options is responsible for the different technology and integration choices to make this dream a reality.
While the role of a Data Architect can vary depending on the organization, it usually encompasses some of these components:
- Help define the right choice of technology stack
- Define the data structures and…
