Automating sustainability insights for investors: The interaction between sustainability specialists and data engineers at Clarity AI
This article has been written in cooperation with Oscar Gomez but sadly, medium does not allow us to set a co-author.
Introduction
Oscar: OK, Joseda, the reader’s initial expectation is to understand the purpose of our writing, correct?
Joseda: The main idea of this article is to make people understand how two different profiles can work together and give value to the customers in an efficient way.
But before we deep dive into it, we should provide some context on what we do at Clarity AI. Notice that the bold sentences are written by Joseda and the ones in italics by Oscar.
Who are we?
Joseda: I, Joseda, one of the Data Engineers at Clarity AI, and have been working for the company for almost three years. At the beginning I only worked at the EU Taxonomy squad but recently I have been moved to another squad and given support to other squads. Most of the engineers work with other engineers and a product manager, just a few of us have the opportunity to work with a more academic/theoretical profile or with a Data Science team.
Oscar: I am Oscar, and I have been working at Clarity AI for nearly five years! As a Sustainability Expert, my daily tasks involve developing methodologies to evaluate companies’ sustainability, constructing and coding these methodologies, staying updated with the latest regulations from official institutions. Like Joseda, I work daily not only with Product Managers and Data Engineers, but also with Account Executives and Product Specialists.
Some context about Clarity AI
Oscar: I can take this one. So, Clarity AI is a sustainable finance tech startup with 300+ employees. We provide a SaaS that allows investors to better understand how sustainable are the companies and governments in which they invest in.
In particular, Joseda and I belong to the same squad. Our squad is responsible for building a product that allows investors to assess how sustainable the revenue is of companies according to the EU Taxonomy, which is an official sustainability framework defined by the European Union. We analyze the data of +2,000 business activities for +40,000 companies.
To understand it better, a typical use case could be an investor uploading a portfolio of 500 companies to our platform to realize that only 10% of the weight of the portfolio can be considered sustainable.
How is our team formed?
Joseda: Our tech squad is composed of around fifteen individuals, including sustainability specialists, data engineers (DE), backend engineers, frontend engineers, designers, an engineering manager and a product manager.
The Product Research and Innovation (PRI) team consists on five individuals with a profile similar to Oscar in charge of developing the methodologies and understand the regulation we work on.
Today, we want to deep dive into the interaction between the sustainability specialists and data engineers. But before it, Oscar, can you start with sharing which are the main responsibilities of a sustainability specialist? Or in other words, how do you bring value to Clarity AI?
Value proposition and responsibility
Oscar: Sure. Our main responsibility is to come up with the methodology.
We are also in contact with the investors that use our product and planify collections of company reported data, develop models to assess the different aspects of the regulation with the data we have, enrich the models with different provider data…
An average day can change a lot, some of the typical tasks of a sustainability specialist can be:
- Sketch in a python notebook the methodology to assess a new metric required by a regulator
- Find proxies/estimates to assess whether companies from a specific sector meet the sustainability requirements established by the EU Taxonomy
- Read the update into the regulations and summary so other member in the team can understand the update easily
Joseda, can you very briefly describe your main duties as a data engineer? I am sure that people will understand the data engineer role compared to mine.
Joseda: On the other hand, the main responsibilities of a data engineer are the following:
- Ensure that the data can be updated and validated in the required frequency
- Ensure best development practices are applied in each project
- Follow architectural guidelines during the development
- Challenge PRI team (Oscar in my case) to simplify the methodology and be able to split in small incremental pieces
- Being able to give PRI team the tools they need to work efficiently, this includes automating certain tasks and provide useful reports and notifications
Now that we’ve covered that, let’s focus on our interactions. These include:
- Ensure that the data produced is what the client need and complies with the data quality standards from Clarity AI
- Validate the generated data during a data release
- Automate the methodologies for the calculations
- Assist each other in case of a client request
- Evolve and simplify the methodologies together
Daily interaction
Joseda: Normally, both DE and PRI member work in the same team delivering the same product. One has the methodology in mind and the product features and the other has the technical expertise to be able to implement the methodology and automatize the generation.
As part of this process we work more closely together when we start with a methodology and start refining it, the first idea is not always the one that is delivered and through several iterations we manage to build what our client needs. This cooperation does not end when the solution is implemented, we also need each other when we have some data changes and we need to validate the data we want to deliver or when we want to adjust something in the methodology and we have to run tests and try different adjustments in the algorithms.
When PRI receives a client request, they may require the help of a DE. With advanced data wrangling skills, a Data Engineer can aid in providing results or evaluating hypotheses more efficiently than PRI could independently.
Frictions
Oscar: Some of the typical issues that we experience are:
- One team being a bottleneck for the other. For example, the sustainability experts might have prepared a methodology to assess a specific group of companies to implement into the product but the data engineers might not have the bandwidth to focus on it.
- Lack of context regarding the other time period.
- Using different terminology for identical concepts.
- We each believe we understand one another, but in reality, we’re conceiving different ideas.
Communication style
Oscar: Our main interactions are done by slack and some weekly meetings. We tried to be as asynchronous as we can since we work with people from different time-zones, so most of our processes are automated and we are able to know if the process finished correctly or not looking at a notification channel in slack.
If we have to read a report with the execution results, the notification will contain the link to download the notebook rendered with the information we need to confirm that the algorithm behaves as we expect.
A small piece of advice
Oscar: Based on our experience we think this is like any other team, it takes time to be able to work together effectively. Get to know each other first and understand each other’s limitations and strengths, this will help you collaborate and be able to support the other team members when they need your skills.
Joseda: Let me provide an example for clarity. During a substantial data point update in our company, we needed to assess the differences in module coverage and data point variations for specific companies.
While PRI was able to accomplish this, we created a report to simplify the process and allow for comparisons between two distinct executions. As a result, each new data release comes with automatically generated insights.
DE worked with PRI in constructing the report, incorporating all critical information necessary for validation. While this information may not bear significance from an engineering perspective, it holds value for the sustainabilty expert.
I think we have already enough to wrap up with this article. We should not forget to give kudos to Aadil Maan and Alex Ewerlöf, as we get the idea to write this from their article Staff Engineer & TPM: A conversation about tech’s two most unique IC roles.
Oscar: Thank you for reading! If you want to contact us, you can do it through LinkedIn. This is Joseda’s and this one is Oscar’s.
