An Intro to API Key Management
Last month, we hosted our first webinar on API key management. CryptoMove CEO Michael Burshteyn covered the recent trends in API keys and secrets management, including an overview of current challenges and solutions as well as an exclusive sneak peek into our product Tholos, the first key vault with advanced moving target data protection. You can watch the recording here: https://youtu.be/i_a7w5O-ej0. Read on for a transcript of the discussion.
[0:00:00] Mike here from Cryptomove and we’re doing this webinar regarding variety of different topics. So, we want to go through a deep dive on secrets management, API key management, encryption keys and other types of secrets related to applications and application security. And what are the challenges that are emerging these days and various approaches to them. Particularly around this idea of threat modeling and looking at the risk around how API keys and secrets are used and so we’re going to start with that, with some of the issues there. Then get into some of the best practices and some of these questions and then go over a deep dive into how moving target defense can be applied to API key management, [0:01:00] secrets management and that sort of model approaches around that and then finally walk through a beta product that we’re building at our company at CryptoMove around API key management and go through the questions that we have received. So, to start out with, the interesting kind of thing when we speak with security teams around API key secrets configuration files and certificates is that there are more every time you poke your head up and look around there seems to be more and more. And so there is this issue of proliferation and we started digging in to try to understand why that is and a few of the things that came up in our surveys of security teams and they kind of relate to a variety of things.
So, one is just in [0:02:00] the way of developing cloud native applications. Often, applications are broken up into a variety of services components and that means that those various services have to speak with each other and require a series of secrets in order to do that in an authenticated manner versus non cloud native development more monolithic applications just don’t have that. Then you can sort of multiply that across multiple clouds. There is this interesting idea around the transition between user identity and machine identity. So for a long time the key identity markers in an organization especially large enterprises have been people and business users and that’s how identity has been managed. But more and more as processes are being automated with things like robotic process automation or just [0:03:00] API integrations, the machines themselves are taking on jobs and roles that people traditionally did. And so, those machines now have to have identities and those involved keys and secrets.
DevOps is an interesting one that kind of at least triples the volume, just because you have to have a set for your development environment, your staging environment, your prod environment and just sort of automated deployment integration and those types of things require a ton of secrets and so kind of going down the list the story makes sense with microservices, containers, AI, big data obviously now require more secrets management. IoT is an interesting one, so crypto move, we’ve actually been closely involved in a variety of IoT project where you see literally [0:04:00] thousands of sensors and devices spread out around a smart city for instance, all of which have to have pretty heavy duty secret management rotation.
And the last one that is kind of overlooked is blockchain, but it’s interesting and definitely affects the threat model around secrets because with cryptocurrency or non cryptocurrency related blockchains the way to access them is with private keys and so suddenly private keys can represent hundreds of millions, and billions of dollars and transactions or important information. They can’t be lost, destroyed or exel traded or else there’s serious issues. So that’s kind of this issue around secrets management and then contrast that with what is the traditional approach to key management in enterprises and so that’s something where we want to see sort of the difference, [0:5:00] you know in terms of what is the proliferation of secrets management and secrets due to these various changes in infrastructure and then also what is the traditional approach.
The overview or sort of what we’re going into just a quick recap for people that are just joining or catching up on the video feed is: we’re starting out with secrets management a deep dive, especially from a threat model and risk-based approach. And looking at why there are more API keys and secrets today. And it’s due to changes in infrastructure and in the way software is getting developed. Then we’re going to walk through sort of what the implications of that are and some tools and best practices. And then get into living target defense and how that can be applied to this problem area and go over the threat modeling on that. And ultimately actually walk through some of the beta product [0:06:00] that we’re working on at CryptoMove.
So, with the kind of transformation of infrastructure that’s leading to proliferation of API key and secrets management it’s interesting to contrast that with how many organizations — that are not as familiar with these new infrastructures or that are undergoing transformation into these infrastructure — and how they approach key management and it’s a different approach. It’s very focused on encryption keys, on secrets related to PKI infrastructure, public and private keys, certificates. The idea of API keys and other application-related secrets hasn’t really been looked at too closely. And traditional approaches tend to be hardware based so that’s a dominant kind of HSM hardware security model and we’re seeing more modern approaches where HSMs are used for root of trust [0:07:00] but not necessary for all the keys.
And there is this interesting challenge just around facilitating speed of development and a lot of developers — and we’ll get into how this is happening to companies like Pinterests and Lyft and Square — developers will actually end up going out and developing their own secrets management tools simply for better efficiency and speed of development across teams and not necessarily even security as the driver. And so you see these popping up in different companies.
So briefly, obviously there are events which are resulting from this proliferation of API keys and they tend to fall into a few buckets of developers leaving keys up on GitHub or potentially sharing them via unsecured channels or simply misconfiguring [0:08:00] cloud-based resources and not setting up the appropriate access credentials. And so, when we look at sort of these types of environments one of the things we’re very interested in, and that I think a lot of practitioners are still trying to figure out including us, is what are the most important questions to ask when you look at keys and secrets for applications in an environment to have a good sense of the attack surface and the risk and threat model. And so these are some of the questions here on the slides that we have.
The interesting thing about these questions is that, as we’re developing our products around key and secrets management, these questions essentials drive a lot of the analytics section of that. So when we think about what dashboard we want to display, what visualizations [0:09:00] related to access patterns or key rotation, or anomalous behavior, these are some of the questions that we look at. And we think that if we can come up with a set of 10 to 20 questions then any organization should be able to map out their tax surface and then plug in kind of their threat model. Because there are a variety of different approaches where you might in one organization be worried about one type of threat related to API keys and secrets and another worried about others. And we’re — -A few examples of that may be you know the threats across the spectrum of just insiders in an organization using keys and limiting access between various developers or machine resources to various systems of data.
There are threats related to third parties if related to partners [0:10:00] or other parts of kind of the software supply chain and sometimes those should be less prioritized or more prioritized. I think one of the critical things which needs to be developed in the industry is a way to look at first what is the threat model that this organization cares about and then what are the best practices based on that threat because I think it widely changes.
This is something we’re working on. We’re actually developing a survey methodology to do this and we want to build it into our app. In terms of other interesting projects in this area, so GitHub has taken a really leading role in looking at how developers are using code repository sometimes for things like [0:11:00] an encrypted SSH keys. And they’ve scanned…for a long time they’ve been scanning internal repos for GitHub oauth tokens, but they haven’t really expanded that until recently. And so they just ran a private beta at GitHub and we’ll circulate this link. This was just published about a week ago. It’s very interesting on how they were able to expand their token scanning ability using this library from Intel (it’s called the hyperscan library. And it enabled them to essentially scan millions of repositories on GitHub and find millions of tokens that were unsecured. And then what they do is they basically alert the developer and or alert the service provider.
So if it is a AWS token they would alert AWS that here is an unsecured token to your services. And then different service providers [0:12:00 ] can take various actions based on that. So we thought that was an awesome projects and I think it is now in public beta (t’s been in private beta and just moved over to public beta). It’s just an example of organizations where they’ve ran into secrets management issues and they’ve started to play around with their own and then open source them. These are some interesting tools. I think Keywhiz is probably the most popular and is one of the first that’s built by Square. They use it in production which is kind of cool and it seems to be quite functional and we’ve actually set it up successfully and prototyped it at CryptoMove. So that’s an interesting one. There is also Knox and Confident as well. So, Lyft and Pinterest had these issues.
So, anytime you see a lot of open source [0:13:00] projects like that popping up, you know that’s a lot of people having similar problems and we think there’s opportunity to have a better solution. So, when we look at what does that solution look like we are thinking about what are the critical values to maximize for secrets management. And I think these values are important not necessarily just for software — for instance we’re in product mode — but even for process. So, I think that when we look at it, some of the most mature secrets management processes and kind of policies relate to the lifecycle of the secrets, authentication, being really crisp about identifying machines and not just people who have access to secrets. I’m going to skip the middle one and then jump into reliability, scalability of the infrastructure and whether that infrastructure is in a lot of cases pen and paper [0:14:00] or just communication coordination between teams or software — that scalability and reliability is important. And then also, just being reactive and being able to understand events or anomalous behavior related to secrets.
The middle part is, I think where there isn’t a ton of innovation ultimately secrets kind be thought of as a data, as keys or essentially just strength or series of strength and they are often stored in a database that could be encrypted but they ultimately are a stationery target and that’s where a moving target defense gets really interesting because you can actually take secrets and keys and spread them out and make them a moving target individually fragment them and continually alter their properties whether the encryption envelopes around them or mutate them or move their location and play [0:15:00] sort of a shell game around. So, I think that’s where there is very interesting opportunity for innovation and that’s part of what we’re working on.
In terms of challenges around secrets management this is a list we’ve put together just based on commonalities we see at various enterprises and organizations. And number one is just prioritization. The interesting thing about secrets management is that it’s an inbetween type of issue because it affects the engineering department development and impacts the speed of development efficiency, working in teams and sharing secrets and it’s a security kind of domain issue. And so often times that gets lost in prioritization.
Another issue that we see quite a bit not necessarily with tech companies but in other verticals is that encryption key management is pretty much the only thing that people think about and so [0:16:00] API keys are not just considered and application secrets are not really thought about until fairly late in the cloud transformation maturity cycle.
So here is the maturity model and so basically this looks at the maturity of an organization in five stages and specifically related to how they look at secrets management. So, I think the sweet spot is stage three. A lot of organizations are in stage three where there is cloud infrastructure experimentation, experimentation with containers and microservices and actual agile software development processes. And these keys are proliferating and they are starting to become managed or the problem is starting to bubble up.
I think the inflexion point is also at stage three because at stage one organizations, ultimately there is this culture change that needs to happen and so basically any start up [0:17:00] that is kind of looking at emerging technology one of the best practices that is emerging are these maturity models. And when we talk with other DevOps startups especially and developer tools infrastructure startups, the common thread is that technology cannot solve any issues if absent some sort of cultural foundation that makes that technology relevant especially emerging technology. And that’s why there is an innovation curve, and not all organizations are the earliest adopters. and so it’s really important for an organization to map itself out too on where it fits on the maturity model and obviously different maturity levels for different technologies but that just something that we’ve been working on and that goes into prioritization.
Other common challenges are in a hybrid infrastructures, so multiple infrastructures [0:18:00] around keys and secrets can be challenging across data centers or public cloud or private cloud environments, scalability is one of the biggest and just having that clustering ability and administrative support. We see a lot of CLI and API based secrets management tools which are great but there is a lack of UX and UI tools here. And we have heard from development teams, that they actually do like to have a good UI for going into this TN secrets management tool.
Root of trust which we talked about a little bit so this is where HSMs still have a critical role to play. So ultimately there is always a turtle down at the bottom. There is really interesting talk from Netflix….so Netflix gave this talk on how they bootstrap secrets and identity in their cloud infrastructural development environment. And the general gist of the problem is ultimately there is always a secret to start [0:19:00] the secrets management tool. And if that secret — -at least there is a certain threat model where that secret can compromise the entire system. And so there is interesting challenges and approaches to that whether using Shamir’s secrets sharing to split the secrets in the beginning or potentially using hardware and actually having HSMs anchor the secrets. And there is another kind-of very interesting approach or at least methodology of framework around this called crypto anchoring, where you essentially think about if you use some sort of HSM or crypto to anchor a secret or a piece of data before it can be accessed, then you essentially force the attacker to operate within your environment. And so that can be leveraged to create a strong root of trust but [0:20:00] it’s very interesting and I do think HSMs have a role to play there.
In terms of ownership of the problem, is a challenge it relates to prioritization. We kind of talked about it is the DevOps security. Discovering the secrets is a huge issue. So we’ve been running our beta at CryptoMove for a few weeks and one of the things we noticed is when we onboard people the first thing they do is they leave our app and look for their secrets. And that is painful to watch because we’ve incurred different folders and different cloud accounts and it’s mess. And so what we’re thinking is that that discovery friction point is huge and the more that can be done to reduce that friction the better adoption would be of secrets management in an organization. So we’re working on different types of survey methodology, the automated tools to do that. And have certain ideas about intercepting common workflows to automatically kind of pull secrets [0:21:00] into a system.
And then finally, behavior tracking, I think that’s huge. It’s interesting to think about how much time is spent on vulnerability prioritization. And imagine taking a little bit of that mental energy and putting that into mapping out how secrets are being used and accessed in an organization can paint probably an interesting picture for a threat model. And so those are just some of the management challenges that we see. In terms of moving target defense, so now we’re little over halfway through and going to get into how moving target defense works and how it applies to KN secrets management. And so basically moving target defense is a broader and bigger issue, or bigger type of field security [0:22:00] than just around API key secrets.
In general, the idea of moving target defense is to take the asymmetric advantage of adversary and use it against them. And I think, it’s one of the most heavily researched areas of security and we have a blog, post where we kind of a catalog our favorite papers around moving target defense and measurement of it but it’s actually one of the lesser commercialized solutions. And so with moving target defense — -You have a lot of academic research, a lot of military research and not a lot of cross application. And that’s just because it’s an emerging field and one of the big questions around moving target defense is — okay, if we can take the defense infrastructure and make it dynamic. So today if you have a non-moving target defense or a stationary target, the longer an attacker can hide [0:23:00] and study it the easier it is for them to pull off their attack. And so if you can make that target move around, shift its properties, suddenly time is no longer an advantage it’s actually a disadvantage. And this is a fundamental concept that can reorient the asymmetry between attackers and defenders. So that’s sort of the basic theory.
And you can look at moving target defense through game theory perspective and you can actually model it. There are certain studies that have been done (papers) and MIT actually has a very good sort of comprehensive guide to all moving target defense that is. But you can do a quantitative analysis of how making the…kind of reconfiguring the attack surface actually affects the difficulty of attack. And model things out like response time [0:24:00] and success probability and we’ll circulate some of this work. We’ve actually done specific modeling of that with CryptoMove. So there is a lot of general moving target defense modeling out there. And then we actually did a model, that basically does goes through what happens when you split data up and move data around and how does that affect the entropy of storage relative to keeping data as a stationery target.
One interesting thing about moving target defense, it can be applied across the stacks. You can have application level moving target defense and ASLR is the most common it’s adverse spaced layout randomization but there are other approaches. There is network level moving target defense which started out essentially with IP rotation, [0:25:00] and rotating network access. And now there’s CryptoMove and that’s where we’ve been focusing our innovation and where we have our focus.
And it’s been heavily researched in the military and various federal agencies. So a few years back there was an executive commission on cybersecurity in terms of what are the top next generation strategies. And so there is a difference they wanted to look at: not just the top tools such as artificial intelligence or the top technologies for security like clouds and things like that, but they wanted to look at strategies. Are there ways we can actually change how we do security rather than just doing security in similar ways but with better tools? And moving target defense was sort of one of the top recommendations which was a way [0:26:00] to re-imagine how we approach security all together.
And so that was that executive report and it led to projects since the Department of Homeland Security where they’ve actually funded quite a bit of moving target defense research including CryptoMove. So we’ve been fortunate to work with them as well as the Airforce where they’ve been doing network level moving target defense research for quite a while. And they actually sprung out a technology that they worked on that is now a startup. And there are few other startups doing moving target defense. There is one working on polifimofic limits called PoliVerse. There is a company called Morphisec which is doing moving target defense related to endpoint protection. And CryptoNight and XT which is related to network protection. And then obviously there is CryptoMove where we are focused on data and secured storage of keeping secrets and eventually [0:27:00] other types of data.
So that kind of gets us into the…towards of the backend of the webinar. So I want to make sure that I leave time for the questions that I have that people sent it. But I am going to just walk through a quick demo and I am going to go from the UI all the way down to the CLI level.
So basically if we look at CryptoMove we get in, we authenticate and we are able to basically see the secrets and what does the secret look like. And have the ability to upload secrets and mask them or not. You can name them, so if you have your Stripe API key or you know Bot3. There is a variety of ways to manage, the key is whether you want to transparently generate the keys [0:28:00] with the recipe or have a CryptoMove do them. Or bring your own key even and utilize APIs for that. When the keys go up either programmatically or via the UI, what we’re doing is we’re splitting them, mutating, encrypting each key, padding it, creating…basically scattering it across our data store and let’s say we want to go in.
We have this concept around classification to show various levels of sensitive of different keys and secrets. And you can think about custom tags like dev or prod or you know tag something with Google or GCP or something like that. Those are some other ways you can start to categorize keys and secrets and then once you get into them you know, it’s great to be able to modify the description, so this key is for a bot we created to talk to with our customers that need extra help. [0:29:00] And you can do things like modify the expiration date which is really helpful for automated rotation. So one of the big challenges people have is they have key rotation policies and they are not automatically implemented. So it would be great to just have a policy that says all keys rotate at every week for this type of service and then have that happen.
You can display your secret in our system which actually pulls it back and recombines it. And then there is this interesting sharing idea, so if I wanted to share my key with my team mate they could have access to [0:30:00] it and I could revoke it and there is administrators that could do that as well.
And then we have this concept of versions as well where it’s important to be able to create new versions and rotate your keys. And so that’s kind of the basic functionality and then under the hood the question is, okay there are a variety of questions around who are the users, how are they provisioned, what do they have access to. And then how is the moving target defense component taking care of and then what services are connected whether it’s your identity and authentication services, various cloud services or things like that.
We are working on some analytics as well where you want to map the key attack surface and some of the questions that we are talking about actually lead to some interesting insights around how keys are expiring and who is accessing them and what are the classification that the keys are in. We’re working with some vulnerability assessment experts. Actually one of advisers is focused on this area from the data science perspective to try to figure out what are the most impactful visualizations in the analytics to map the key attack surface.
Here we have our APIs under the hood, so obviously the idea is everything has to be programmatic as well. So for every UI, every interaction [0:31:00] there is the ability to call the API and do things like send the key in, modify the expiration date. And our data users are already essentially working on integrations and all the dynamic key generation rotation with different third party services and the API is going to be a big component of that.
And then lastly, I’ll just walk through what’s going a little deeper under the hood. So in the CLI of CyproMove, we have quite a bit of functionality in our command line tool. As we were installed for years basically working on this secure data store, and have gotten to the point where you can put data in, get data out, modify its functionality, make the data a moving target. We’ve prototyped this in alpha with a variety of systems [0:32:00] in public cloud, private cloud, even installed CryptoMove onto drones with some of our customers and have used CryptoMove in application like live streaming video, so that’s something interesting and this is just showing you our help commands and help menus in the CLI.
Another kind of under the hood, sort of view of how CryptoMove works, here’s a prototype we created for file. We’re basically, we’re monitoring file system interaction, this is something we worked on in the early R&D where we can intercept the file system calls, so suck those files out and the file size is good as zero and they are being split up and moving round the CryptoMove datastore. And when you open them it synced to the Linux users so it’s transparent able to bring them back together. This can thought of essentially as a replacement for endpoint encryption. And there was recently an [0:33:00] SSD encryption issue with the voker and this could be a nice software alternative to hardware encryption. But we’re prototyping that for later.
But really just to show kind of the under the hood stuff. So on the left you have the debugging log, those are key end data fragments moving around. On the right we can take this PDF, we’re going to split it a 100 ways, two copies and move it about 28,800 times a day — that’s about once every 3 seconds. The client will split it up and move it around and do things like that. And then it will start, just basically moving through the data store randomly. And if we want to look at what it looks like on the file system, that was a key an API key or any type of data these are just encrypted data fragment. And so they are moving around, they are each encrypted separately and they are constantly changing and getting re-encrypted. So even just trying to open them, it doesn’t [0:34:00] necessarily show what the key is when you open it. And so that’s just making it more and more difficult from the threat model of somebody who has infrastructure access to get everything and bring it back together and it can reduce the attack surface too because you can limit, using crypto anchoring, you can kind of limit the access so everyone just have access to their own key.
Just a comparison of the side by side, so the entropy over ten seconds this is that moving target defense concept where even with just ten seconds going by the color coding shows how the data fragments are different and how much they have moved and changed just in that small amount of time. And so this means that basically you can create a choke point and then with good authentication and almost behavior detection for access on the keys, you can kind of choke off the attacks that would rely on grabbing the keys [0:35:00] from the database or any type of lower level infrastructure attacks. And if you combine that with good sort of secrets management best practices as we spoke about with rotation and having enterprise secrets management tool, that combination of the sort of frontend of the threat model and the backend of the threat model can cover a lot of ground. So that’s really where we are at CryptoMove.
In terms of just some of the questions that we’re asked and sent over email…so one of the questions that a lot of people wanted to know about is moving target defense, how can you actually measure its advantage? Does it make me 5% more secure? 10%? 100%? 1000%? That’s a really interesting question and it’s hard to answer. But you can actually start digging into the math and the risk based model. We think it depends on the type of organization and what the landscape of that organization is. But there is [0:36:00] a way to model out the risk reduction of moving target defense applied to various systems.
But the most important piece of it is not necessarily like a linear percentage increase but this idea of actually changing the curve entirely and taking it from a situation where attackers have the advantage due to time to where time is actually a disadvantage can essentially mean that detection is not as necessary because even if you don’t detect the attacker, they just can’t accomplish their objective as easily.
There is a common question that gets asked which is: what are some examples of attacks where MTB would have made a difference? And some of those attacks can be looked at where people have left API keys, [0:37:00] their secrets out as a stationery target, where you might see attacks like with OneLogin they actually had data that was encrypted that got decrypted because an attack came in, studied the system over time. The most recent one that was big in the news was around SSD encryption being kind of compromised. They used inside channels and that impacted the BitLocker. The intells, [inaudible 0:37:28] vulnerabilities, any type of those lower level infrastructural attacks. Sometimes we speak with organizations where they are just concerns about insiders and how those insiders at partners or club providers might look at infrastructure that’s on the moving target side. I think on the key management side just a share organizational tool, just not even having moving target defense under the hood, just having keys managed can resolve a lot of issues [0:38:00] related to key mismanagement and auto rotation and things like that.
So, I am just going through all of these questions since this is a recording and obviously anyone in the audience can shoot in additional questions. But I have a list here to just keep going. So, one common question is whether CryptoMove the product we’re building is called Tholos, is it a primer in the cloud. And so our answer is that it’s cloud native. That means we have a SaaS hosted version and that’s how we’re running our data and that’s how sometimes we work in pilots and trials. But it can also be put on-prem or into a private cloud environment. And so that just means that the environment should be a cloud native environment where you can take advantage of services based architecture because that’s really important for a skim.
And, there was an interesting [0:39:00] question around how does CryptoMove fit into a multicloud world. And so this is a common question. I think multi cloud infrastructure is one of the reasons for keys and secret proliferation and that fragmentation of infrastructure makes it hard to manage keys and secrets in one place across an organization. And I think this is where a tool, like the ones we’re building like some of these ones built in some of this homerolled one that we looked at like Knox and confident. The reason they are getting built is because there has to be something across multiple clouds. And we think this is a huge value of CryptoMove and our Tholos tool or any secrets management tool as not being native to any one of the clouds but across all of it. We’re partnered up with the Amazon and we’ll be at AWS re:Invent in a couple [0:40:00] of weeks actually featured as one of their top ten startups that they’re featuring in the startup central area. And we have also facilitated at top fortune 100 enterprises at the same time and we partnered up with people at Google and discussed challenges Google customers are facing, Google cloud key and secrets management.
It’s definitely important to be multi cloud in terms of some other questions from email, basically include what are the top enterprise key related workflows? And so this is something we discussed a little earlier just related around what do security teams really do around keys and secrets? And so I think it’s all about these major objectives. So here management [0:41:00] lifecycle management rotation, authentication, storage and that’s one where there hasn’t really been innovation. We’ve just been throwing them on hardware boxes and then in databases. It’s kind of be going backwards actually. Scale and auto access.
And then one of the top questions is just around APIs and will CryptoMove and this Tholos product be accessible problematically is a huge question. And so the answer is yes, and we’re planning to put a lot of examples and documentation on GitHub. We actually just started our GitHub account and you know we’re going to be doing some interesting things there. We’re actually going to open source a pretty large project that we’ve been working on for years while we were installed. So we’ll be open sourcing a programming language that we actually developed inhouse at CryptoMove. And so we’ll have API’s [0:42:00] and all that.
That’s kind of the overview of this webinar. So I think we covered all the ground we wanted to, we talked through what are keys and secrets proliferating? How would you look at an attack surface and model it? Can you actually start to survey the organization and come up with models even before implementing any tool at all? Is there just a way to raise the level of intelligence on what that threat model looks like? How does moving target apply in this world on the backend, on the storage level and also what are the best practices to build a really sustainable enterprise key and secrets solution? And where is CryptoMove in that and what does our roadmap around that look like? [0:42:55]
So that’s all from me. This is Mike from CryptoMove, thanks everyone for listening.
