My First 200 Days on the Financial Crimes ML Team
Hello! I want to share my learnings as a senior ML Engineer on a completely new team (formed in 2020) under the Legal branch of Square. This is inspired by and maybe more organized than my previous “First 100 Days” article.
Anyway, here’s a breakdown of my observations into the 5 W’s + How.
Who
As of August 2021, my immediate team is comprised of my lead/manager, 4 machine learning modelers, 1 machine learning engineer (myself), 1 data scientist, and 1 data analyst. You might be wondering what each of these roles mean. Here’s my breakdown and interpretation, with partial job descriptions pulled from Square career pages.
I like to think of these roles on a spectrum, labeled 1 through 9. Let’s say 1 means super analysis heavy and 9 means super engineering heavy and 5 is in between.
Data Scientist
Independently research questions related to our most important product experiences and produce recommendations for peers and leadership.
Design, run, and analyze experiments with the goal of optimizing acquisition, activation, and retention measurements
I would say a data scientist is a 1–3 on my spectrum depending on the company. Their day to day is extremely analysis heavy. They know their way around tools that can push them towards answers to various product questions such as how do different factors affect churn.
Machine Learning Modeler
Build machine learning/deep learning models that detect risk (credit or fraud) activity in real time across our Seller’s ecosystem consisting of payments, banking and debit card products.
You will leverage experimentation mindset along with state-of-the-art algorithms to drive down false positives, collaborate on new product features to drive losses down and explore new datasets (including 3rd party data) to engineer new features for risk models.
The ML modeler fits right around a 5 on my spectrum. Their day to day is filled with all sorts of tasks in pursuit of building and deploying the best models. This involves both analysis of the data and performance of the model as well as creating new features in existing systems.
Machine Learning Engineer
Design elegant ML pipelines and services as well as prototype new approaches and productionize solutions at scale for our 36+ million active users
Create a world class platform for training, hosting and maintaining ML models
Apply ML and engineering best practices to shape how Cash App develops, tests and maintains ML-platform solutions
Help design and execute our long-term Machine Learning strategy across Cash App
The ML Engineer sits on an 7 or 8 on my spectrum. The day to day activities involve both modeling as well as engineering, with some analysis every now and then. A unique aspect is that the ML engineer may spend time planning for the long term ML Ops strategy & solutions.
Data Analyst
Work with partners to make decisions across the organization by applying descriptive and predictive analytics where it will have material impact
Apply a diverse set of tactics including statistics and quantitative reasoning to solve problems as well as research and produce relevant insights
Manage and solve complex, cross-functional problems with data driven analysis
The data analyst is kind of a tangential role to the three ML oriented roles above. They underline the work that the team does by providing the ETLs and data the team needs. Their work is super cross functional as team members rely on their domain knowledge and their data expertise.
What
We are under the Compliance organization so much of our machine learning efforts have been towards risk mitigation with respect to compliance. This work involves both CashApp as well as Seller, the two main business units at Square.
The bulk of the focus for myself and many of my team members has been on transaction monitoring. We aim to use our models to catch money laundering, trafficking, sanctions related offenses, and other illicit activities.
75% of my day to day efforts is dedicated to the long term ML strategy that our team will take and building out integrations and solutions towards these strategies. Much of the questions I think about are ML ops related such as “which ML model hosting solution should we choose” and “how can I run experiments with the models that I build”. I then go about building out the integration or solution.
One such example is our integration with the in house built model hosting and serving solution. Square is in the midst of transitioning to using external services such as Sagemaker for model hosting but until then, many ML teams rely on this internal model hosting and serving solution. I came to the decision with my team that we also need to rely on this solution until the ML Platform team at Square makes Sagemaker an option.
We have deployed our new Transaction Monitoring models onto this in house solution and plan to deploy a few more Customer Risk Rating models soon.
25% of my time is dedicated to developing models and deploying them to production. This would include initial data exploration and feature engineering, selecting and trying out different model algorithms, hyperparameter tuning my models, and deploying my best model out into production to serve a business need.
Currently, I’m in the process of productionizing a Transaction Monitoring model. What this entails is first building out preliminary models in our hosted Jupyter notebooks. Then I use Google’s AI Platform for bulk training and hyperparameter tuning. I also go through a special model governance committee because we serve compliance needs. Lastly, I’d deploy to our in house model hosting and serving solution.
Where
Our team is currently remote and will be in the foreseeable future, fully remote. Square and Jack Dorsey have been ahead of the crowd in terms of adopting a remote first culture.
However, the offices are still there. I go into the SF office maybe once every other week.
When
My workday usually starts at 10am. I acknowledge that I am very privileged and get to start my workday very comfortably at 10.
I usually take 1–2 hours to cook lunch and eat the food.
I normally end my workday around 5–6pm, depending on if I finish early or not.
Why
Square gets fined if parts of the business are not compliant with various rules and regulations. Our team is there to ensure we are compliant and do not get fined.
How
We are allowed full autonomy when doing our jobs, with very little micromanagement. For the bigger projects, there may be a “directly responsible individual” (DRI) who will ask for status updates and ensure the project meets deadlines. If you are the DRI, you’ll need to pseudo manage your teammates in addition to your own work. Aside from that, you’re on your own in terms of how you work.
If you’re interested in working at Square, please take a look at our Careers Page!
Thanks for reading!
