Erik Costlow Of Azul On What We Must Do To Create Nationally Secure And Resilient Supply Chains

An Interview With David Leichner

Know what you produce. Combine the record of what you got with what you did to track what you delivered. Make sure someone can trace back to what you received.

The cascading logistical problems caused by the pandemic and the war in Eastern Europe, have made securing a reliable supply chain a national imperative. In addition, severe cyberattacks like the highly publicized Colonial pipeline attack, have brought supply chain cybersecurity into the limelight. So what must manufacturers and policymakers do to ensure that we have secure and resilient supply chains? In this interview series, we are talking to business leaders who can share insights from their experiences about how we can address these challenges. As a part of this series, I had the pleasure of interviewing Erik Costlow.

Erik Costlow is a software security expert with extensive Java experience. He handles the security of Azul’s JVMs that operate at peak speed while making security easier and better for all. Erik was the principal product manager in Oracle focused on security of Java 8, joining at the height of hacks and departing after a two-year absence of zero-day vulnerabilities. During that time, he learned the details of Java at both a corporate/commercial and community level. He also assisted Turbonomic’s product management team in the data center/cloud performance automation. Erik also led product management for Fortify static code analyzer, a tool that helps developers find and fix vulnerabilities in custom source code. Erik has also published several developer courses through Packt Publishing on data analysis, statistics, and cryptography.

Thank you so much for joining us in this interview series! Before we dig in, our readers would like to get to know you. Can you tell us a bit about how you grew up?

While I think the usual part is to talk about love of computers and when we started programming, probably the more interesting part starts in college. I was studying programming but also at one of two colleges with a circus, so I was also juggling, unicycling, and performing. Also the more I learned about computers and how they worked and were connected, the more I started seeing the corners that people would cut — how you could make computers do something or give you information that wasn’t supposed to happen but could when you knew what to do. Now years later with more technical and business experience, it’s led me to a way of taking complex topics and just making them fun for people.

Can you share the most interesting story that happened to you since you began your career?

With over a decade of application security work, a lot of the stories are things I can’t say and they’re interesting for reasons that make me wish I didn’t actually know the story. One interesting story that I can tell is about the perception of security issues. At a hacker conference there was a competition to exploit software, they were offering a prize to break into a piece of software that I handled. I wasn’t at the conference but it’s public so that’s fine, I wanted to see the result. Minutes before the session, the group who said they could hack it couldn’t get their exploit working, so they bowed out. Instead of saying that “this one piece of software is the only one that wasn’t hacked” based on the original rules, the conference just took it out and said that every piece of software that was tested was hacked. I think this is interesting because it shows how groups change the rules to generate the attention and headlines that they want to see.

You are a successful leader. Which three character traits do you think were most instrumental to your success? Can you please share a story or example for each?

First, focus on what you want to do and drive your ability to do it. Early in my career I wanted to do programming but the college counselors told me not to because I didn’t take calculus. I pointed out that most programmers don’t use that so why was it a requirement, and learned the skills I needed anyways. This got me my career start. Second, adjust based on what you learn. Originally I thought that CTOs and software executives were the best programmers, but it turns out that’s wrong. The leaders are the storytellers who understand fundamentals and can guide a vision while understanding the way that this vision can become real. This is why I switched from development to consulting and ultimately product management. Third, choose something to be world class at but still maintain general-purpose skills. For me my focus is application security, specifically on Java. If groups need help with certain aspects of security, they can get those anywhere. For the niche of Java security, things needing this specialty find their way to me.

Are you working on any exciting new projects now? How do you think that will help people?

I’m working on Azul Vulnerability Detection, a new SaaS product that continuously detects known security vulnerabilities that exist in Java applications and components. It tells you what code you have, what code you run, and whether it is vulnerable. It’s exciting because it’s the only solution that focuses on detecting Java vulnerabilities in production, which is the critical end step of the software supply chain. This will help people who run Java applications learn about security risk in their applications without becoming security experts. It makes security a byproduct of running your software. It takes the knowledge from the last decade of application security and puts it into the same JVM that people use to run their software, making it possible, automatable, and repeatable at scale.

Ok super. Thank you for all that. Let’s now shift to the main focus of our interview. In order to ensure that we are all on the same page let’s begin with some simple definitions. What does the term “supply chain” encompass?

Supply chain just means “the software stuff you get and provide.” The easy way to look at it is that almost everyone is a consumer and a producer. For Java applications, Maven Central is a producer of open source libraries and developers are consumers of those libraries. Some developers write libraries (as a producer) that they publish back into Maven (which is the consumer). For an organization’s custom software, they are the consumer of the Maven libraries and the producer of custom code to create an application: they produce this application to serve their customers. In the vendor space, companies are producers of applications but they just have this big loop inside to create the software. The place where the software runs is the final consumer, it doesn’t produce anything. The question for that consumer is, “where did all this software come from” and ideally you can trace it back.

Can you help articulate what the weaknesses are in our current supply chain systems?

Certainly there is weakness, and a lot of it relates to vulnerabilities in third party software. According to Gartner, by 2025, 45% of organizations worldwide will have experienced attacks on their software supply chains, which is a 3x increase from 2021. We believe that vulnerabilities in components and libraries play a role in that.

Specifically, there’s almost no tracking or validation going on at all. In a typical supply chain you have a list of inputs, processes that those inputs went through, and the output produced. In software we kind of have some of this but it’s very light. It’s really difficult to trace anything back and everything is designed to ship software to the ultimate consumer as fast as possible. There’s very little ability to look back from that consumer and see where it came from and what happened — even if you get an SBOM, how do you know it’s correct?

Can you help define what a nationally secure and resilient supply chain would look like?

The first step is getting the ability to track what’s in the software that’s used. CISA is starting to drive this with SBOMs (software bill of materials) but we need to establish what’s collected for different parts of code, like what’s tracked for custom code. Ultimately the goal is to trace back a level of provenance — when my supply chain tells me that I’ve received a particular component that came from a place like Maven central, how can I verify that this is what I actually received.

My particular expertise is in cybersecurity so I’m particularly passionate about this topic. Can you share some examples of recent and notable cyber attacks against our supply chain? Why do you think these attacks were so significant?

Last year a friend of mine, Matt Austin, attacked Microsoft teams. He opened the Teams binary to look at their internal components, then registered that component in a public repository with a slightly higher version. In supply chain terms it was a fake product claiming to be from Microsoft but actually a knock-off made by someone else. The supply chain unfortunately didn’t recognize the actual owner and thought they were the same. In the name of security, builds Teams took his component because the name matched but the version was higher. Within a week he had harvested the internal names to prove the attack and reported it to Microsoft. They worked together to fix it but also kicked his fake item off GitHub and npm. This is a nice person doing security work and helping the affected party so it’s not a major notable incident but this is the type of the validation in supply chains.

What would you recommend for the government or for tech leaders to do to improve supply chain cybersecurity?

I’d love to give any other answer, but we need regulations and fines for malpractice. If there’s no incentive to improve security and no liability or major downside of companies doing a poor job then they’ll continue doing a poor job. In this case the government is mandating certain types of security testing to procure software or SaaS, which causes companies to do them.

Ok, thank you. Here is the main question of our interview. What are the “5 Things We Must Do To Create Nationally Secure And Resilient Supply Chains” and why?

To generally under-complicate it, we can set a baseline set of recommendations:

• Know what you receive. Record the ability to track what comes in to you supple chain.
• Know what you produce. Combine the record of what you got with what you did to track what you delivered. Make sure someone can trace back to what you received.
• Help identify what’s at risk. The industry tracks many items with CVEs, so enable yourselves or others to accurately recognize where known vulnerable items are.
• Patch or defend your portion. When a security flaw is found, do adequate patching of what you produce. This could be putting a new version into the supply chain. Otherwise provide a defense if you can’t patch — you’re responsible for whatever is in your possession at your point of the supply chain.
• Avoid custom knowledge. Real supply chains are large and cross companies. Someone looking back over the supply chain should be able to track and understand items and they shouldn’t need special training just for your section. You can be complicated but you should be consistent and clear.

Are there other ideas or considerations that should encourage us to reimagine our supply chain?

We really just need basic tracking and the ability to look back and validate that what we think is there is actually what’s there. At the same time, we need a way to look and tell if any of the components we’ve received are vulnerable, indicating we need to bring a newer fixed version through the supply chain.

You are a person of great influence. If you could inspire a movement that would bring the most amount of good to the most amount of people, what would that be? You never know what your idea can trigger. :-)

There are a lot of ways we can go here and many good ways to improve the world. I generally stick to the things I know like technology, where I think a goal should be to improve the things that we have instead of re-inventing everything over and over, and to make improvement that matter not things that make it worse but drive small revenue. For example the supply chain solutions should understand supply chains, they shouldn’t re-invent them with some strange web3 crypto token thing.

How can our readers further follow your work online?

I should always tweet more over on @costlow but I mostly write articles over on InfoQ, a site for architects and developers.

This was very inspiring and informative. Thank you so much for the time you spent with this interview!

About The Interviewer: David Leichner is a veteran of the Israeli high-tech industry with significant experience in the areas of cyber and security, enterprise software and communications. At Cybellum, a leading provider of Product Security Lifecycle Management, David is responsible for creating and executing the marketing strategy and managing the global marketing team that forms the foundation for Cybellum’s product and market penetration. Prior to Cybellum, David was CMO at SQream and VP Sales and Marketing at endpoint protection vendor, Cynet. David is a member of the Board of Trustees of the Jerusalem Technology College. He holds a BA in Information Systems Management and an MBA in International Business from the City University of New York.