the interview: Stephanie Sher
December 2020
Stephanie started her tech career in software, as an early employee at NYC-based infrastructure monitoring company Datadog. She has been a product manager in robotics/manufacturing automation as well as led growth in the ML devtool space, and now serves as GM at a stealth startup. She enjoys organizing communities of founders, executives, ML engineers/data engineers/data scientists at some of the world’s leading deeptech companies, and also advises and invests in applied AI/ML companies.
What do you view as the main challenges to continued progress in AI/ML?
The biggest challenges I see can be loosely classified as technical and organizational. Technically speaking, there is a lack of education and experience around successfully deploying models in production — things around data quality, monitoring health of models in production, bugs in the ETL pipeline, drift in data distribution, outlier identification. Companies — to the detriment of their bottom line and with the consequence of missed opportunity — keep degraded models up and running. Organizationally, it’s the willingness and capacity of existing teams to support the ML function — and for ML teams to understand what success looks like in an industry context.
As co-organizer of Full Stack Deep Learning, what do you see as the biggest impediments to broader adoption of ML within organizations? What trends do you see that are encouraging?
Teams will face timeboxed opportunities to reach a common understanding of potential, urgency, and scope of their ML needs. To really answer the question of how pursuing a methodology could impact their bottom line, and what the tradeoffs are, and getting everyone on board with that. ML is slightly less straightforward than say, traditional software engineering, and so there will be a tricky bit of having comprehensive organizational alignment around what it takes to have a model succeed in production, and how much of that is worth investing in alongside legacy systems.
An encouraging trend: Tooling is a hot space right now, and I believe the competition will yield a few winners that can really accelerate and improve the process of deploying in production in a way that aligns with business incentives.
Which part of the ML toolchain do you think is the most painful or most lacking in supporting infrastructure?
At the moment, most blockers still lie in data preparation. 90% of what is preventing people from successfully deploying ML in production is related to data management. The next part of the chain is training tools, but those are pretty good already; the space is nearly saturated. But pretty soon we’ll see a shift towards the rest of the ML toolchain: monitoring the health and integrity of models in production; understanding why we’re getting the results we’re getting; being able to identify issues early and react to them in a timely manner.
What lessons did you learn during your time at Datadog that you think are transferable to the ML Ops ecosystem? What similarities and differences do you see between APM and monitoring of ML workflows?
So much, and nothing at all. There are of course similarities in the sense of knowing how a SaaS B2B tool comes into being, how to guide product development alongside initial market research, how to distribute effectively. But while we do see a lot of the same terminology being thrown around, ie monitoring and observability — at the same time, I think it is a very different technology that looks more like the discipline of software engineering integrated with the art of machine learning rather than just SWE, which on its own is a bit more straightforward.
In terms of the market, something I learned at Datadog is that you can’t guess based on first principles how the ecosystem will evolve. For example, people like to write about Datadog’s “strategy” in the beginning but it was much less intrinsically directed, and perhaps had more to do with landscape and positioning, than people like to think. So I suppose the “lesson” here is mostly an excitement to see how the different players in the ecosystem approach the MLOps challenge, and how users react to different tools that surface.
One thing I’ll call out here: Datadog has a world-class solutions engineering team that has been crucial to the company’s success. I expect to see more of this from companies, whether APM or ML offerings — diligent investment into highly competent onboarding and implementation support.
Where do you think we are in the development of the ML Ops market? What is holding back growth, if anything?
I moved to the bay area in 2017, at a time when hundreds of millions of dollars were being poured into moonshot ideas. It just didn’t make sense to me. People who *were* talking sense, in my opinion, like Rodney Brooks and such, were demonized and classified as pessimists. Recently, with more tangible proof points in the form of companies getting funding and succeeding or failing, and then layering on recent things like instability in global markets and politics, the tech industry has had to face reality a bit and focus on pragmatic aspects like “hey, how are we going to actually make these things work? Will this add value to our customers in a tangible way that inspires them to pay us money?” And so I’m glad to have witnessed what has been great progress in a more practical direction, especially in the devtools space. One perhaps obvious thing holding back growth is talent — as always there is a dearth of technical talent to effectively utilize the tools that we’re starting to build. I’m encouraged by trends of researchers leveling up to become functioning SWEs, and engineers with domain expertise learning more about the machine learning side of things — it will take this sort of interdisciplinary thinking for machine learning to grow healthily. I’ll give a shoutout to Full Stack Deep Learning here — we have alumni across verticals like healthcare, insurance, robotics, education, all enriching their skillsets to both specialize in a focus area and solidify their knowledge of machine learning.
You have been involved at the forefront of several emerging industries. What research or technical developments in the industry are you the most excited about as you think about the next 2–5 years?
I’m intrigued by this new area of augmented intelligence, defined by Gartner as “a design pattern for a human-centered partnership model of people and AI working together to enhance cognitive performance, including learning, decision-making, and new experiences.” It sounds.. magical, almost. But there are little tangible improvements that we can make to how we consume, store, and process information, and we’re starting to see companies try to wrangle that. I’m interested to see where that goes.
In the MLOps world, ideally we’ll one day be able to see models in production that are updating in real-time. We’re not quite there just yet due to ramped up operational infrastructure needs (more computing power, better monitoring/alerting, attention to failover responsibility) that most companies aren’t operationally prepared for yet, but as with APM monitoring and alerting, I hope we’ll soon see latency improvements across the board, driving towards more timely and effective decision-making.
I must add here that I’m equally excited about applications of data and machine learning outside the tech industry. I’m seeing a shift of interest towards more effective utilization of data with the goal of better and more timely decision-making in industries that have traditionally been less technical or commercially viable, in arenas like the environment and food supply and microfinances and such. Hopefully the money follows the momentum.
What work are you most proud of?
I’m proud of every student that has come through the Full Stack Deep Learning program. I think it takes a special kind of chutzpah to explore the boundaries of your field, and to do that outside of your normal delineated work hours. In the FSDL community, I’ve seen people complete our curriculum while taking on full time jobs, internalize and leverage our curriculum to land jobs at, for example, NASA’s Frontier Development Lab, and push each other within the community to work through both tactical and strategic, organizational and technical problems while trying to deploy machine learning in production at an industry level.
For myself, I’m not sure “proud” is the word I want to use, but there are two experiences that I’m especially grateful for. First, being the second marketing hire at Datadog and demoing our product across the United States and Europe alongside AWS as it scaled up, back in 2014 — that was an incredible opportunity and something I learned a lot from. It was very exciting to be showing the world what was coming their way — this incredibly pragmatic tool that would streamline their workflows — I remember telling people it’d give them more time to spend with their friends and family, haha. Second, because I care a lot about advancement in software/AI/tech in general and even moreso in conjunction with community and education — I’m grateful to Josh and Sergey and Pieter to have brought me onto the Full Stack Deep Learning team, to push forward a curriculum that teaches ML practitioners how to deploy machine learning in production. Again I’m not sure “proud” is the right word, but working with them on this community has definitely been, and continues to be, a most rewarding and worthwhile endeavor, full stop.
There is a third piece of work that’s very exciting to me: I’m working with some friends in the space on an initiative to make hands-on experience with robotics and machine learning more accessible. It is still in early stages, but I’m very excited for its potential to put research into practice. Troubleshooting, debugging, monitoring live progress.. For me that’s where the fun begins.
What advice would you give to women building their careers in AI/ML?
Eat well, exercise regularly, put in the work. Flywheel effects will occur with an accumulation of good bits in place, and it’d serve us all well to be prepared when the Cambrian explosion hits. Exciting things are happening in our field, so keep learning, keep pushing. And reach out; I’d be excited to meet you. :)
