Data Science: real jobs beyond the buzz
It has been now almost 4 years since I joined Criteo and I still keep a vivid memory of my first day on the job. Hired to lead a team of 4 Data Scientists and being “given access to one of the world’s biggest datasets, and the computing power to exploit it” (as per the job desc) I was under a mix of excitement and puzzle. Although the hiring process went well enough, I wondered what such a trade required as skillset. The contours of Big Data, Data Science, Advanced Analytics were definitely hazy to me and I wasn’t quite sure to what extent my telco engineering background, an MBA and a professional-services work experience would be an asset or a liability as I embarked on this new endeavor.
Today, the team has grown to nearly 50 Data Science professionals coming from a variety of backgrounds. Although Data Science jobs now become more commonplace, there is still lots of unclarity about what those entail. This post presents how we think about those in Criteo’s Product team, hoping to help the reader get a clearer picture of what’s the reality beyond the haze.
First, it is helpful to present what is it Criteo does. Criteo is a company that manages digital ad budgets on behalf of clients. Criteo is specially good at optimizing these budgets’ allocation for performance. The optimization is very much propelled by a suite of AI-powered software components which, altogether, make the Criteo solution. Among those components one can find: recommender systems, bidders, dynamic creative optimizations systems and others. These components are tuned and combined in specific ways to cater to specific client needs or objectives. Certain of those combinations are then framed as products which then are proposed to clients.
Within this frame of reference, the role of Data Scientists and the two profiles of Data Science professionals we have in the team: ‘Product analyst’ and ‘Algorithm specialist’ can be made explicit more easily.
Here below I will present these roles and their specificities along 4 aspects of the job: their respective mission, the activities they entail, the tools they required and the scientific foundations they leverage. Let there be light!
The first difference we make between Product analysts and Algorithms specialists is about the expectations we have about each role; that it their Mission. Product analysts are expected to find ways to optimize the performance of a product, typically by surfacing key insights when Algorithm specialists are expected to build models that which will enrich and enhance the capabilities of the company. Those models are typically not directly tied to a specific product and are used to serve various needs.
These different expectations have far-reaching consequences in terms of organization of work, activities, skillset sought after and success criteria that I detail below.
From afar, both populations can be seen to have very similar activities: both access data from a repository, apply analytical approaches and reasonings and make accessible to a less analytical audience the most relevant part of their work. However, different missions mean different activities and engagement model:
Product analysts typically work within a feature-team-like structure bringing together all required talents (often a combination of Engineering, Product Management and Analytical ones) around a shared objective. In that context a Product analyst’s work will typically include:
- Building the relevant data sets from design to production; so as to be able to understand the performance of the product and highlight improvement areas
- Creating the analytical frameworks (simulations, optimization scripts, performance metrics) that will allow getting an offline estimate of the identified business opportunities
- Defining, driving and analyzing results of A/B tests to assess whether online performance is in line with offline simulations
Beyond those typical activities, Product analysts play also a key role in ensuring a smooth roll-out of the optimizations passed, collaborating with our global commercial teams.
Algorithm specialists tend to work in a more distant fashion from their clients. The crux of their work is to invent and build tailored (typically ML-based) solution fitting Criteo’s specific needs; an activity that requires less frequent collaboration from other skillsets and longer research/test/learn cycles. An Algorithm specialist work will typically include:
- Reviewing goals, success metrics and expected timelines and milestones for the projects they take part in; to ensure there is shared visibility & understanding on the features and deadlines that are required and to ensure the necessary tradeoffs are made as early as possible
- Researching approaches in their field of expertise; to keep abreast of new developments
- Building solutions: from creating data pipelines to testing approaches, to improving them and to building the delivery method (e.g. web app)
- Ensuring management of their solutions’ lifecycle (e.g. feature requests, models rebasing etc.)
Beyond those typical activities, Data Scientists are involved in the global Criteo AI/ML/DS community; ensuring continued emulation and ensuring spread of knowledge that can be reused efficiently throughout the organization.
Although this is the most common argument I hear during interviews when one tries to express a preference for a job over another; the question of the tools required to do the job is not discriminating at all.
Product analysts and Algorithm specialists alike leverage ETL tools, notebooks (typically using R or Python), visualization tools and versioning ones. Similarly both populations will tend to leverage the same open-source libraries or company-specific ones.
The main difference between the two populations is probably to be found in the use of statistical models that are readily available in packages or libraries. When Product analysts are expected to use such package proficiently (that is, being able to explain the results they yield and stress test them) but as a means to an end, Algorithm specialists will use those packages as a baseline to start their modeling and app building effort from.
Criteo’s business being data-rich by design, Product analysts need to use proficiently statistical or probability concepts to carry their work; soon enough one gets very much at ease with sampling or hypothesis testing techniques. Indeed, with tens of Terabytes of fresh data everyday to start from, we have too many ways to get to the insights we want. Certain approaches will be fast, some will be slow; some will be very precise, some will be less so; some will allow to get a comprehensive perspective along many dimensions, and some will only give you a number. Those questions, which can safely be overlooked when the cost of an analysis is close to 0 (think of a pivot table on Excel) become critical as one needs to wrestle with such amount of data to single out discreet but powerful insights. And when the result of your analysis is to impact a global system that connects with 1.4B users, you want to make sure your recommendation is sound!
Algorithm specialists often have a deeper statistical training than Product analysts but their distinct expertise really lies in their command of modeling be it to craft Statistical models of ML ones; being able to balance the pros and cons of one class of model vs the other and able to explicit what the tradeoff of choosing one class over the other implies, now and for the future iterations of their solution. Such expertise, we’ve found, typically requires having dived deep in the underlying mathematical or ML concepts which explains why Algorithm specialists commonly have had longer trainings than their Product analyst colleagues.
A different skill set but a shared mindset!
Working at Criteo as a Product analyst or an Algorithm specialist, large quantities of data, innovation and a stimulating environment are pretty much a given. Similarly, the emphasis on the job will be on being able to craft the right approaches, understand how the solution you’re building improve the value we bring to clients and understand how your solution will fit within the whole Criteo and online ecosystem. Agility at finding the best approach across various model classes and flavors is the core capability for Algorithm specialists. Dexterity at crafting the best approach to surface killer insights across large and diverse sets of data is the one for Data Analysts. Both jobs have much in common and complement themselves very well to find new and innovative ways of doing business and influence Criteo’s development.
My four years in the team have taught me that such ability at crafting the right approach for the right problem is not the privilege of any specific curriculum but more so the consequence of an interest for technology married with a curiosity to understand how things work and meshed with a drive to test, learn and find new ways to improve upon what’s existing. That is this latter inclination that has made me pass day 1 on the job and kept me energized ever since!
If that sounds like a career you’d feel energized by too, I’d be happy to hear from you! Any question? Feel free to comment below!