The Curious Case of the declarative Azure Bicep template “not declarative”…

Photo by @olloweb on Unsplash

Can an IaC (Infrastructure as Code) tool that uses declarative syntax behaves as an imperative tool?

Unexpectedly, the answer is ‘yes’ :-)

In this post, I want to share with you how I spent 2 weeks in order to deploy some resources in Azure Cloud for a bizarre behavior of the newborn Microsoft Azure DLS.

But let’s start from the beginning….

Infrastructure as Code and Azure Bicep

In Flowe, we are big fans of Automation and we always follow the “Automate Everything” principle.

Personally, I really love declarative IaC tools and in my Job experiences, I used a lot of them (Terraform, CloudFormation, AWS CDK, AWS SAM, Google Cloud Deployment Manager,…).

Some months ago we decided to give a try to the newborn Azure Bicep DSL.

Azure Bicep is the new IaC tool provided by Microsoft that after a period in preview became ‘production-ready’ and now Microsoft supports it 100%.

According to the Microsoft Azure documentation:

“Bicep is a domain-specific language (DSL) that uses a declarative syntax to deploy Azure resources. In a Bicep file, you define the infrastructure you want to deploy to Azure, and then use that file throughout the development lifecycle to repeatedly deploy your infrastructure. Your resources are deployed in a consistent manner.”

Note: as we will see later, the keyword here is ‘declarative’

When I started to learn it, I discovered it provides a lot of interesting features:

• Simple syntax (similar to Terraform)
• Modularity (for code reuse and DRY principle)
• Support for parameters files (for different environments deployment)
• Support for conditional deployment
• Support for iterative loops
• Read and use existing cloud resources to create new resources

Under the hood, Bicep templates compile to ARM templates, so anything you can do with ARM you can do with Bicep.

So, after I discussed its potential with my teammates, we decided to start using it to deploy our Azure resources.

Let’s try it!

We decide to try it with two different Function Apps:

1. a ‘simple’ Function App in a dedicated App Service Plan

2. a more ‘complex’ Function App with a dedicated App Service Plan, VNET Integration, Access Restrictions, and Storage Account firewall

The ‘simple’ Function App

Our simple Function App was composed of the following resources:

In order to create it with Bicep, we needed:

• an ‘App Service Plan’ (which can host multiple Function App)
• a ‘Storage Account’ (dedicated to our Function App)
• a ‘Function App’ (using the previous ‘App Service Plan’ and ‘Storage Account’)

Creating a Bicep template for these resources was really simple.

Assuming you need to deploy it in an existing Azure Resource Group, you need just to:

• create the ‘App Service Plan’ resource:

• create a ‘Storage Account’:

• compose the connectionString needed to connect the Function App to its Storage Account:

• create the simple ‘Function App’:

That’s it.

So we put all the previous code in a unique Bicep template (https://github.com/thecillu/azure-bicep-mistery/blob/main/main.bicep) and we were ready to test it.

We validated the Bicep template with the following AZ CLI dry-run command:

az deployment group what-if --name simpleFunctionDeploy --resource-group myResourceGroup --template-file main.bicep

and finally, we deployed it in Azure:

az deployment group create --name simpleFunctionDeploy --resource-group myResourceGroup --template-file main.bicep

When the deployment finished we found the new resources created in Azure and we were ready to upload our ZIP file containing our dotnet functions.

SO FAR SO GOOD, with a simple Function App architecture Bicep did its Job.

Let’s play hard!

For the second Bicep test, we made our Function App architecture a bit more complex:

We added to this complex App:

• Access restriction for both the Storage Account and the Function App: in this way our resources were accessible only from configured IPs/Subnets
• VNET integration for the Function App: in this way our function could communicate in a secure way with other Azure resources in our VNET/Subnet

Note: enabling the VNET Integration, the communication between our Function App and its Storage Account pass through the integrated Subnet, and the Storage Account firewall were configured in order to accept traffic just from this Subnet

To create this kind of resources with Bicep, we complicated a bit our code:

• reusing the previous ‘AppService Plan’:

• reading from Azure our existing VNET (‘my-existing-vnet’) and Subnet (‘my-existing-subnet’):

• creating a Storage Account with Firewall and Network ACLs:

• creating the Function App with VNET Integration and Access Restrictions:

Finally, we put all the new code into a single Bicep file (https://github.com/thecillu/azure-bicep-mistery/blob/main/main-complex.bicep) and we were ready to create our resources:

az deployment group create --name complexFunctionDeploy --resource-group myResourceGroup --template-file main-complex.bicep

The Deployment finished without errors, telling us all our resources were created with success.

Wonderful! (we thought…)

At this point we did the last part of the process, we built our Function App Zip and we deployed it into the new Function App, waiting until the Function App restarted…

…and here the problem started…

Where are our functions?

After the Zip deployment, we took a look at our Function App to see proudly the result of our Job…

…..but we didn’t find nothings…..

Instead of the list of our dotnet functions, we found a mysterious message:

Azure Functions runtime is unreachable

What did it mean?

We didn’t have any idea, so we started to search for the error on the Microsoft documentation and we found an interesting article:

Troubleshoot error: Azure Functions Runtime is unreachable

This article provided a list of reasons and possible solutions for our error.

We tried to apply all the Microsoft suggestions, without success :-(

Took a look at the Bicep created resources we also found other strange things:

• The Function App didn’t have in the keys section any master or default key
• The Storage Account didn’t contain the folder created from the Function App during the first deployment

At this point, as Azure Enterprise Customer we decided to open a ticket to receive help from Azure Technical Support.

The Ping-Pong communication with the Azure Support Team

A really kind Azure Support Engineer contacted me the day after opening the ticket.

He started asking me more details about our error and then started a ping/pong communication of several days, 20 emails, and 1 call, where the engineer basically asked me (several times) to:

• check the credentials to Access the Storage account from the Function App
• check both the Function App and the Storage Account Firewall configurations in order to understand if they were configured correctly
• Trying to access Kudu console (the App Service Administrative console) in order to verify the status of the deployed functions, the logs, and the communication between the Function App and the Storage Account (using the tcpping command)
• Delete and recreate all the resources

No suggested actions resolved the issue :-(

In particular, the tcping command did from the Kudu console confirmed we didn’t have any network issue between Function App and its Storage Account:

D:\home>tcpping mycomplexstorageaccount.blob.core.windows.net
Connected to mycomplexstorageaccount.blob.core.windows.net:80, time taken: 68ms
Connected to mycomplexstorageaccount.blob.core.windows.net:80, time taken: <1ms
Connected to mycomplexstorageaccount.blob.core.windows.net:80, time taken: <1ms
Connected to mycomplexstorageaccount.blob.core.windows.net:80, time taken: <1ms
Complete: 4/4 successful attempts (100%). Average success time: 17ms

So, why Function App was not able to access its Storage Account?

EUREKA!

At this time, after almost 2 weeks, we had lost hope because all the attempts were unsuccessful.

Finally one day, during a call with a colleague of another technical team, I told him about my strange problem and suddenly he told me something which opened my mind:

“I remember I had a similar problem with other Azure resources. If I remember well I resolved it creating my resources before without firewalls; then I modified my bicep and run it again enabling the firewalls into the template”

WHAT???

This sentence could mean just one thing: in some way, during the deployment, there is an initial timeframe where Bicep is not able to able to well understand/manage the configured network rules that permit the communication between the Function App and its Storage Account through the VNET integration.

If this theory was true, there wasn’t any chance to deploy the resources and the related firewall in the same Bicep file :-(

So, what to do?

The Solution: using Bicep like an imperative Tool

Due to the fact it sounded horrible to me deploy the Function App using a 2-steps deployment and 2 different Bicep templates, the idea was to use Bicep as an Imperative tool, modifying the template in order to:

1. Define a first time the Storage Account resource without Firewall enabled
2. Define all the other resources (Function App included)
3. Define the Storage Account again; this time with the Firewall enabled and putting a dependency of this resource with the Function App resource

To code this idea, we introduced the module ‘storage-account.bicep’ which permitted us to define 2 times the Storage Account resource observing the DRY principle:

The module took in input the storage account name and a bool which enable/disable the firewall on the provided subnets array. This module permitted us to define the Storage Account resource twice, a first time disabling the firewall and a second time enabling it:

We put all the new code into a single Bicep file (https://github.com/thecillu/azure-bicep-mistery/blob/main/main-complex-fixed.bicep) and we launched it

az deployment group create --name complexFunctionDeploy --resource-group myResourceGroup --template-file main-complex-fixed.bicep

when ready, we deployed our Dotnet Functions Zip, and finally, the magic happen!

Our Function App was able to communicate with its Storage Account through the VNET integration and our functions were ready to run.

Conclusion

I think Microsoft Azure should take a look at this bizarre behavior of the newborn Bicep IaC tool.

If you use a declarative DSL you don’t want to configure it as an imperative one.

Btw, I still suspect the problem could be not strictly related to Bicep but could be a transactional problem in the Azure Resouce Manager which manages the ARM templates. Since under the hood, Bicep templates compile to ARM templates, I think we could replicate the issues defining our resources using directly ARM templates.

We still believe Bicep is really a great tool; we are using it for our different Azure resources deployment, and we never had other strange behaviors.

Maybe it’s not 100% mature, but however, we feel confident to suggest you as an IaC tool for your Azure resources (mostly if the alternative is to use the complicated syntax of the ARM Templates :-) ).

We hope this article will help somebody who found a similar problem using Bicep ;-)

Do you want to hear more about Flowe? Look at all the open positions!