How different data teams interact
Fernando Vendrameto is a Data Engineer for the Latam Enterprise and Message BU.
Vitor Prado is a Data Engineer Leader for Latam Enterprise and Message BU.
Currently both work with SMS and WhatsApp in Latin America for Sinch in the same team but used to work in different teams.
We have been using Airflow for a while now and BigQuery for analytics. Airflow is the main tool for our integration and data collection from some of the different companies that Sinch has acquired. This empowers our analysts to create reports, deliver them to customers, and facilitate billing, especially in Latin America, which is a key focus for us in the Latam Enterprise & Message BU. Also, our team is actively seeking external plans and generating crucial reports that allow customers to analyze their performance across messages, chatbots, and other products.
Our main objective revolves around creating a centralized data lake that consolidates information from diverse sources and Sinch’s product portfolio. With such a vast array of offerings, our goal is to establish a hub where we can generate insights for other teams. To achieve this, we’re adopting new stacks and tools. One such tool is the Orchestration tool, which we’re collaborating on with the entire team. We rely on Airflow to orchestrate our data pipelines, operating separate instances to integrate with different data sources and feed the data lake.
While we focus on internal evaluations of our product performance and client satisfaction, it’s worth noting that our day-to-day responsibilities also involve collecting data from various sources, ensuring its cleanliness and reliability, and providing other Sinchers with access to this valuable information. While the Latam Enterprise and Message BU team utilizes BigQuery, the Applications BU team relies on Snowflake.
The Enterprise and Message BU faces a significant challenge in managing the volume of message platforms in the region. Processing enormous amounts of data is a daily task, such as handling 40 million SMS and 10 million WhatsApp messages. In contrast, the Applications team, responsible for new architecture and products, isn’t currently facing volume-related hurdles as we focus on scaling for the future.
At Sinch, even though we operate in separate teams and pursue different projects, collaboration is at the heart of everything we do. We frequently engage in discussions, exploring opportunities for collaboration and knowledge sharing. In the realm of data orchestration, we actively seek ways to support and assist one another.
Our most pressing short-term challenge is integrating diverse data types from the companies we’ve acquired. Establishing a centralized solution and generating reports for all these products in one place is crucial. What sets our integration process apart is our direct interaction with departments that developers may not typically engage with, such as global finance and business teams, who provide valuable insights into our clients’ needs.
The principle of “Win Together” resonates within our team. Our achievements thus far have only been possible due to the dedicated individuals who made it happen, despite the numerous challenges we encountered. We maintain ongoing collaboration with other data teams, consistently working together to create new possibilities.
Being a data engineer at Sinch is primarily about simplicity, regardless of your area of focus. We stay grounded, prioritizing tasks that add value without unnecessarily overcomplicating things.
