Injecting Faults in Azure Key Vault using Azure Chaos Studio — Rest API
Azure Key Vault helps one to store,
Secrets Management like tokens, passwords, certificates, API keys, Key Management like encryption keys used to encrypt your data and Certificate Management like TLS/SSL certificates.
It helps one to centralize, store securely, monitor its usage (who accesses and how), and easily integrates with other Azure services. Azure Key Vault benefits many organizations.
Let us assume if the Key Vault access is denied? What will happen to the application(s)? Will the application fail totally or will it fallback to other region/service? How the customer gets impacted if the access is denied? Will this trigger many errors, or add latencies? How to handle it gracefully?
How to test this scenario?
Yes, the answer is using Azure Chaos Studio!
In this blog, I will cover how to inject fault in Key Vault from Azure Chaos Studio using REST API methods. This blog will cover step by step activities, right from enabling targets with necessary capabilities, create experiments on the target where the fault needs to be injected, add an appropriate role, and invoke the experiment.
Note: I built these demos using Postman. The same can be done using other REST API based tools and can be automated.
In my experiment svdkvchaos is the key vault name and chaosrg is the resource group name. Subscription id masked with a dummy one, please use your sub id to run the test successfully.
The Chaos Studio experiment’s name is chaoskeyvaultAPI.
How this fault works?
Blocks all network access to a Key Vault by temporarily modifying the Key Vault network rules, preventing an application dependent on the Key Vault from accessing secrets, keys, and/or certificates. If the Key Vault allows access to all networks, this is changed to only allow access from selected networks with no virtual networks in the allowed list at the start of the fault and returned to allowing access to all networks at the end of the fault duration. If they Key Vault is set to only allow access from selected networks, any virtual networks in the allowed list are removed at the start of the fault and restored at the end of the fault duration.
Pre-requisite’s before building a Chaos Experiment in injecting fault on Azure Key Vault:
The target Key Vault cannot have any firewall rules and must not be set to allow Azure services to bypass the firewall. If the target Key Vault is set to only allow access from selected networks, there must be at least one virtual network rule. The Key Vault cannot be in recover mode.
Reference: Chaos Studio fault and action library | Microsoft Docs
Let us look at building the experiment step by step,
1. Create a Key Vault in a region supported by Azure Chaos Studio (the below Key Vault is created in West US)
QuickStart — Create an Azure Key Vault with the Azure portal | Microsoft Docs — refer this doc for creating a KV.
2. Execute this command to understand what capabilities Azure Chaos Studio has for Key Vault
GET
Outcome:
Currently (at the time of composing this blog) Key vault supports only DenyAccess feature (DenyAccess-1.0).
{
“value”: [
{
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos/providers/Microsoft.Chaos/targets/microsoft-keyvault/capabilities/DenyAccess-1.0”,
“name”: “DenyAccess-1.0”,
“systemData”: {
“createdAt”: “2022–07–02T18:16:52.2893512+00:00”,
“lastModifiedAt”: “2022–07–02T18:16:52.2893512+00:00”,
“createdByType”: “User”
},
“type”: “Microsoft.Chaos/targets/capabilities”,
“properties”: {
“publisher”: “microsoft”,
“targetType”: “keyvault”,
“description”: “”,
“parametersSchema”: “https://schema-tc.westus.chaos-prod.azure.com/targetTypes/Microsoft-KeyVault/capabilityTypes/DenyAccess-1.0/parametersSchema.json",
“urn”: “urn:csci:microsoft:keyVault:denyAccess/1.0”
}
}
],
“nextLink”: “”
}
3. To enable the Key Vault to run Chaos Experiments on it run the below command
PUT
Body(JSON)
{
“properties”: {}
}
Outcome:
{
“location”: “westus”,
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/Microsoft.KeyVault/vaults/svdkvchaos/providers/Microsoft.Chaos/targets/Microsoft-KeyVault”,
“name”: “Microsoft-KeyVault”,
“systemData”: {
“createdAt”: “2022–07–04T07:32:53.8069487+00:00”,
“lastModifiedAt”: “2022–07–04T07:32:53.8069487+00:00”,
“createdByType”: “User”
},
“type”: “Microsoft.Chaos/targets”,
“properties”: {}
}
4. If you want to disable the Key Vault target, please run the below command (this should be done post the experiment),
DELETE
5. To enable the capability available in Key Vault (DenyAccess-1.0) run the below command
PUT
Body (JSON)
{
“properties”: {}
}
Outcome:
{
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos/providers/Microsoft.Chaos/targets/microsoft-keyvault/capabilities/DenyAccess-1.0”,
“name”: “DenyAccess-1.0”,
“systemData”: {
“createdAt”: “2022–07–04T09:17:49.487094+00:00”,
“lastModifiedAt”: “2022–07–04T09:17:49.487094+00:00”,
“createdByType”: “User”
},
“type”: “Microsoft.Chaos/targets/capabilities”,
“properties”: {
“publisher”: “microsoft”,
“targetType”: “keyvault”,
“description”: “”,
“parametersSchema”: “https://schema-tc.westus.chaos-prod.azure.com/targetTypes/Microsoft-KeyVault/capabilityTypes/DenyAccess-1.0/parametersSchema.json",
“urn”: “urn:csci:microsoft:keyVault:denyAccess/1.0”
}
}
6. Create an experiment that will deny access to Key Vault for 10 minutes continuously
PUT
Body (JSON)
{
“type”: “Microsoft.Chaos/experiments”,
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/Microsoft.Chaos/experiments/chaoskeyvault”,
“name”: “chaoskeyvaultAPI”,
“identity”: {
“type”: “SystemAssigned”
},
“location”: “westus”,
“properties”: {
“selectors”: [
{
“type”: “List”,
“id”: “8b4f8f9b-b4e7–419d-933a-104d44a92e45”,
“targets”: [
{
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos/providers/Microsoft.Chaos/targets/microsoft-keyvault”,
“type”: “ChaosTarget”
}
]
}
],
“steps”: [
{
“name”: “Step 1”,
“branches”: [
{
“name”: “Branch 1”,
“actions”: [
{
“type”: “continuous”,
“selectorId”: “8b4f8f9b-b4e7–419d-933a-104d44a92e45”,
“duration”: “PT10M”,
“parameters”: [],
“name”: “urn:csci:microsoft:keyVault:denyAccess/1.0”
}
]
}
]
}
]
}
}
Outcome:
{
“type”: “Microsoft.Chaos/experiments”,
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/Microsoft.Chaos/experiments/chaoskeyvaultAPI”,
“name”: “chaoskeyvaultAPI”,
“systemData”: {
“createdAt”: “2022–07–04T09:50:26.327866+00:00”,
“lastModifiedAt”: “2022–07–04T09:50:26.327866+00:00”,
“createdByType”: “User”
},
“identity”: {
“type”: “SystemAssigned”,
“principalId”: “ha3586ff-1268–4370–969b-f33ce59a155p”,
“tenantId”: “85f988bf-43f0–40af-91xy-2d7cd012fb47”
},
“location”: “westus”,
“properties”: {
“selectors”: [
{
“type”: “List”,
“id”: “8b4f8f9b-b4e7–419d-933a-104d44a92e45”,
“targets”: [
{
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos/providers/Microsoft.Chaos/targets/microsoft-keyvault”,
“type”: “ChaosTarget”
}
]
}
],
“steps”: [
{
“name”: “Step 1”,
“branches”: [
{
“name”: “Branch 1”,
“actions”: [
{
“type”: “continuous”,
“selectorId”: “8b4f8f9b-b4e7–419d-933a-104d44a92e45”,
“duration”: “PT10M”,
“parameters”: [],
“name”: “urn:csci:microsoft:keyVault:denyAccess/1.0”
}
]
}
]
}
]
}
}
The experiment is created successfully.
The experiment will invoke key vault deny access for continuous 10 min making it unavailable for the applications.
The experiment will be invoked on the given target, which can also be seen on the target resources.
7. For the above experiment create a Key Vault Contributor Role
PUT
{{keyvaultscope}} → /subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos
Body(JSON)
{
“properties”: {
“roleDefinitionId”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/Microsoft.Authorization/roleDefinitions/f25e0fa2-a7c8–4377-a976–54943a77a395”,
“principalId”: “ ha3586ff-1268–4370–969b-f33ce59a155p”
}
}
Outcome:
{
“properties”: {
“roleDefinitionId”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/providers/Microsoft.Authorization/roleDefinitions/f25e0fa2-a7c8–4377-a976–54943a77a395”,
“principalId”: “ha3586ff-1268–4370–969b-f33ce59a155p”,
“scope”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos”,
“createdOn”: “2022–07–05T05:24:58.5782035Z”,
“updatedOn”: “2022–07–05T05:24:59.3438285Z”,
“createdBy”: null,
“updatedBy”: “f8e9b215–302b-4586–92ee-2275660c26fd”
},
“id”: “/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/microsoft.keyvault/vaults/svdkvchaos/providers/Microsoft.Authorization/roleAssignments/9805844c-c53a-11ec-9d64–0242ac120003”,
“type”: “Microsoft.Authorization/roleAssignments”,
“name”: “9805844c-c53a-11ec-9d64–0242ac120003”
}
Chaos Experiment is given a Key Vault Contributor role to run experiments on the key vault
8. Run the Chaos Experiment.
POST
Outcome:
{
“name”: “chaoskeyvaultAPI”,
“statusUrl”: “https://management.azure.com:443/subscriptions/fbe563b4-c548–11ec-9d64–0242ac120002/resourceGroups/chaosrg/providers/Microsoft.Chaos/experiments/chaoskeyvaultAPI/statuses/EAB73BED-5F1B-4B94-A623-FECC4BDAE58C?api-version=2021-09-15-preview"
}
The experiment is invoked, and it is running
Meanwhile you can check the Key Vault and by seeing the metrics, one can understand that the api hits will come down as the requests are denied by the KeyVault in that 10 mins when the experiment is running. One can also verify by connecting Azure Key Vault with their apps.
The experiment ran successfully for 10 min.
The same can be done via Portal in simple steps.
To conclude, this blog describes how to create and run a chaos experiment for Key Vault by Denying access to it end to end using REST API.
Thank you and stay tuned….
Pradip VS
Cloud Solution Architect — Microsoft
