In a World of Machines, What Am I? The Evolution of the Analysts’ Role

We are in a time where real-time performance is the name of the game. Roles once performed manually by analysts have been handed over, in most cases, to either data engineers, or to models developed by data scientists.

The analyst’s role, however, has become ever more important. They are now the ones who can have a clear view of current trends, while also thinking ahead. This is because they can have a close understanding of behaviors as granular as analyzing a single event, and generalize them to have a causal relationship understanding of ongoing statistical trends.

In the following post, I will:

• Go over the evolution of the analyst’s experience in recent years.
• Focus on what the current tech world needs from analysts.
• Give my 2 cents on what an analyst today needs in order to be a valuable asset as an enabler of model optimization.

Pulling away from infrastructure and manual work

Manual labor belongs to the past

Analysts were once committed to manual analysis of data to provide real-time insights for decision makers (or be the decision makers themselves). Having this apparatus was cost-heavy and time-consuming both for the company and for its clients.

The complete switch to models allows a company to:

• Perform real-time analysis 100x faster than an analyst.
• Scale and create products that perform millions of tasks in a split second.

Manual labor is almost no longer needed, and it is also a burden that the analyst today is lucky to be rid of. Riskified’s full-time analysts are not spending their time looking at orders 1 by 1.

Infrastructure and real-time visualization are also going away

An important aspect of the analyst’s life was, and is, in many places, the management of the data itself.

With time, as data schemes and layouts become more complex, the weight of managing this framework shifts more and more to more technical-oriented departments (data engineers, BI).

This could also be true sometimes for real-time data visualizations, as the need for an intermediary between rather-more-simplified KPI requests and data engineers is less noticeable.

So why do you need an analyst? Well, do we really know what’s going on?

What are we asking?

Billions of events, millions of features, thousands of data schemes. It seems like everything anyone in the company ever needs to know is right there, a mere simple query away.

But what do you look for?

1. What metrics should we be looking at to have a qualitative explanation of the behavior we’re seeing?
2. More importantly, in a world with multiple causes for every phenomenon, and with endless cross trends, what is the simple answer?

This is the analyst’s job. Have a complicated understanding, and then simplify it for others.

How are we impacting the answer?

We have a tendency to look at the world as though we’re observing it from the outside, as non-participant observers. But the reality is, every company creates certain settings that change and impact the world.

This is especially true in fraud. Our performance and decisions are constantly evaluated and challenged by fraudsters, looking to understand how to get through our defenses. Our clients want to know whether a string of orders made fast from the same area indicates a fraud attack or is just a local natural phenomena.

How is our performance, and what are we doing wrong?

A good analyst today needs to be able to find where the company’s models are a swing and a miss, where they’re less able to assess the situation accurately. This could be in terms of fraudulent behavior, which isn’t given enough weight, or a legitimate pattern wrongly assessed as fraud by model classification.

An analyst needs to be able to answer the following questions:

a. How do we find pain points robustly and with minimal TTD (time to detect)?

b. How do we translate these pain points to ML language?

What about tomorrow?

Expect the unexpected

The problem with statistics is that it tells you exactly what needed to be done yesterday. Of course, this doesn’t mean we can’t have models that make predictions based on gathered information, but it does mean that analysts have a lot of room to provide insights about what lies ahead. This is especially true in cases where reality is impacted by current company settings (as talked about in the previous section).

Maybe the prime example of this is the evolving nature of fraud, trying to reverse engineer approvals and declines to understand model behavior and find its weak points. Fraudsters are looking at new ways to bypass anti-fraud protections, be it technological or behavioral transitions.

For example, when seeing a rise in account takeovers in a certain industry, we need to analyze current data and understand not only how to stop it, but how fraudsters will change their behavior to try and bypass our actions:

• Will they change proxies?
• Or change payment methods?
• Attack different products?
• Switch merchants?

An analyst needs to know the impact of their actions on the environment, and how it will react, including secondary consequences. These actions can have immediate impacts on performance and revenue.

This graph describes a certain element of a fraud attack over time. Based on the fraudster’s behavior, can you tell when a certain change in settings was made?

Prepare for the long haul

Preparing for the long term means having a strategy for model training. When it comes to model training, one important aspect is quality and unbiased data labeling.

Challenges vary from data that doesn’t necessarily fall into categorical labels, co-dependent features, and recursive bias. For example, in Riskified, orders deemed fraudulent by our models are declined, and we never get feedback on whether they were fraudulent in the form of a chargeback (a customer denying having made a purchase). These declines might later be fed to model training as orders which are fraudulent, even though we can’t be sure.

How do analysts tie in?

Strong and up-to-date expertise on data trends could provide much-needed insights into further model optimization. Analysts who are familiar with the way data “behaves” can tell the story behind it. In our recent example, analysts can analyze the data sets of declined orders to determine which behaviors are more fraud-oriented than others, based on their experience.

Adopt the darkness

Exodus tells the story of Nahshon, who was the first to step into the water, making a path for the Israelites to cross. A big part of the analysts’ job today is to venture into new territories and be the first to explore new grounds and methods. They need to know where to focus and notice anything unusual.

Say, for example, Riskified is entering a new geographical market. We need to answer the following:

1. How will our models do in the new environment?
2. What new features, data sources, or values should our models look at?
3. What tools can help us better evaluate an event?

A company needs analysts to understand the new and the unfamiliar and inspire new ways of thought. We also need to know where NOT to go, and what products or tools simply would not provide good ROI based on the data, but also based on your educated guesses.

So how does an analyst get there?

Embrace languages and technical skills

While we did say earlier that the more technical tasks would gradually shift to other departments, this doesn’t mean that you can neglect your technical skills.

Quite the opposite.

A lot of your research will focus on having the data be reflected in model performance, so you need to know:

• How the models work.
• How features are developed.
• How to show the value of your insights when they are integrated to model behavior.

More and more analysts today need to know coding for their day-to-day, not just SQL.

Keep your feet on the ground

Constantly explore your data. For me, I found the most effective way to stay updated is to handle data tasks once in a while, and explore behaviors and trends as they happen. Another way is to have a method to pay attention to alerting and notification systems that indicate certain trends.

The idea is to make you the person that answers the question of whether something is viable, yes/no, or if it has potential and could make a real contribution to performance.

Keep moving, but stand firm

With the not-so-gradual shift to the age of models, the analyst’s job is becoming less mundane, less repetitive, and much more challenging. Shedding some technical requirements, yet having a much higher technical understanding. Moving away from tactics to strategy.

But always remember that your point of strength is still your understanding of how the data behaves, what questions to ask, and how to answer them.

The future will undoubtedly incorporate more automation and provide more and more tools for analysts to do our job, but we’ll probably need to keep moving forward with our technical skills to keep up.