Growing a Machine Learning Practice Inside a Large Financial Firm
This post was produced in partnership with Capital One.
I recently had the pleasure of speaking with Zachary Hanif, Director in the Center for Machine Learning (C4ML) at Capital One. I was interested in receiving some unique insight on the challenges of developing a machine learning practice within a large financial organization. I’ve written previously about the machine learning best practices at Google and Uber. I was curious to understand how one would go about building a machine learning infrastructure within a firm such as a large regulated bank; following is a summary of our conversation.
Banks have their own set of stringent regulations that companies like Google or Uber are not subjected to. The common stereotype of a large bank is that there is a considerable lag in the adoption of the latest technologies. Prior to joining Capital One, Zachary racked up experience in startup companies developing large scale applied ML infrastructures. With two years under his belt at Capital One, he has been retrospectively quite impressed with the company’s philosophy regarding technology and how proactively they’ve embraced the latest developments.
How do we build a ML center of excellence from scratch?
C4ML started with only a handful of people at the beginning of 2017 and now has over 100. How did it scale so quickly? As Capital One began applying machine learning to more and more facets of the business, C4ML was created as a center of excellence and in-house consultancy to further catalyze the adoption of the technology across the organization, centering on product delivery, innovation, education and research, and partnerships.
To grow the team, C4ML coupled highly specialized internal recruitment with Capital One’s broader “Technology Development Program” (TDP), a two-year intensive rotation training program for recent college grads with CS/CE experience to apply their skills in a C4ML-specific track (the TDP is a top channel for junior engineering and ML talent in the group). The program helps recent grads determine what type of work they’d like to focus on within engineering and ML, while also training them for the specific skills needed at Capital One, as there can often be harsh learning curves when recent grads enter the professional world. To ensure a smooth transition for incoming talent, junior developers are also paired with more senior, experienced developers.
The team targeted majors in computer science, applied mathematics, statistics and physics. C4ML looked for graduates that were strong technically and showed a passion for their craft.
In addition to hiring recent graduates, C4ML has a robust team of experienced professionals who previously worked with problems that required a lot of data. Specific technical environments, for example, involved the use of Spark, High Performance Computing (HPC) or time series analytics. Professionals that were “double-tracked” were ideal candidates; these are professionals who had extensive analytics experience as well as a solid software engineering background.
In their recruitment process, C4ML encourages a diversity of background and thought. While Ph.D. computer scientists are actively sought, the team approaches recruitment with no preconceptions. One of the most unique individuals to be hired was a material scientist. This hire spent his career studying the crystalline properties of steel. The unique perspective that he brought to the team was a high level of rigor, which Hanif sees at critical to success in the field. Different scientific disciplines will have different cultures; in material science, getting an experiment wrong can mean millions of dollars lost.
How do you organize the team?
It would have been impossible to build up the team if co-location was a strict requirement. Today’s companies need location flexibility to attract the best talent. To balance this, C4ML has teams in a few core offices across multiple geographic locations. Members of a project are not distributed, but rather co-located. Specifically, project members are co-located with their internal customers, which are internal lines of business such as credit cards, anti-money laundering (AML) and fraud. The C4ML team members interact with domain expert business and data analysts across each of their internal customer teams. This makes sense in that it is unrealistic for machine learning practitioners to absorb 20 years of human expertise.
Each customer’s internal organization also has technical talent, including data engineers and data modelers. The dual track skillsets of a C4ML member allows them to speak the same language as the data engineering or the data modelers, and for communication to be more efficient without the need for translation between collaborators.
How do you build a robust machine learning infrastructure in a regulated environment?
Model governance is essential as well as timely, accurate, and unbiased peer reviews.
C4ML has highly-specific infrastructure and tools in place to manage human-generated processes at every step. The team built an internal tool that ensures repeatability of experiments; although hyper-parameter is used to identify candidate ML models, they are logged by the tool to ensure repeatability. The team tracks every experiment, and code is versioned in GIT while data is tracked in a custom-built system. This system enables back-testing and tracks data lineage, and is based on an append-only store. Information of how data was selected is also recorded, with the goal of ensuring that tracking and logging is made as transparent and easy as possible.
The system also includes automated statistical testing to monitor model drift over time. The world continues to evolve even after a model is created, meaning that the models can lose their predictive power. This is specifically important in areas like fraud mitigation, where there is a constant arms race between detection and fraudsters seeking to game the system.
Contributing to the ecosystem
C4ML contributes to many open source projects, including Spark, Kafka and Nifi to name a few (Capital One has their own Github page: https://github.com/capitalone). The C4ML team is also spending time on various research tracks and on writing academic papers that will be submitted to conferences throughout the year.
Deep Learning & the Data Intelligence Conference
My main interest is in deep learning developments, so I had to inquire as to how much deep learning is done at C4ML. According to Hanif, there are multiple projects that use a collection of ConvNet, LSTM, custom neural embeddings, reinforcement learning, and graph convolutional networks. The kind of data that are processed span images and time series. With regards to NLP, more traditional methods are being used.
In summary, I’m glad to have gotten a bit of an inside look at how Capital One is approaching technology, especially by proactively and aggressively exploring advanced developments in machine learning. To check out more about their work and research in the space, you can head to Capital One’s Data Intelligence Conference this June, where Zach and team will bring together the broader machine learning community to discuss the latest advancements and research in the field.
