Deploying a .NET Core App to Azure — A Real-World-Example & Problems I’ve run into

Last week I have struggled deploying my web applications from my local development environment into the cloud to get ready for testing with other people. Since I have created a .NET Application it was obvious to choose Microsoft Azure as service provider; and the 130 $ free monthly credits included in my MSDN supported this decision as well. Unfortunately, there are lots of problems I’ve run into and here I want to share my experiences with you.

The Problems

  1. SSH doesn’t wanted my key files
  2. My Web App didn’t event start ( 502.3 Bad Gateway )
  3. Web App can’t connect to SQL Database
  4. Get the wrong URLs from OpenID discovery request
  5. Error 404 — Web app not found.
  6. OpenID discovery still shows the wrong schema

Setup

My setup was quite simple; just a SSL reverse proxy and two ASP.NET applications with a custom domain:

  • SSL Reverse Proxy => nginx & Let’s encrypt running on a Ubuntu VM
  • OpenID Connect Provider => ASP.NET Core running as App Service with a MSSQL database
  • API => ASP.NET 4.5 running as App Service with a DocumentDB

The proxy is configured once manual, but both of the ASP.NET applications are pulled continuously from private repos on GitLab.

1. Creating the resources

First of all, there are two ways of creating resources at Microsoft Azure; using http://portal.azure.com or the PowerShell cmdlets. Since the management portal is often slow I’ve decided to use the PowerShell for initial setup and then modify with the portal.

Creating the Ubuntu VM with PowerShell didn’t worked out of the box, so after several tries I have created it on the web portal.

2. SSL Reverse Proxy

To connecting to the Ubuntu VM I’ve used the SSH client from bash, which is part of the Linux Subsystem for Windows.

Problem 1: SSH doesn’t wanted my key files

This was the first error I’ve run into, when I tried to connect to the Ubuntu VM.

$ ssh <IP> -i /mnt/c/Users/Paul/.ssh/id_rsa
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0700 for '/mnt/c/Users/Paul/.ssh/id_rsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /mnt/c/Users/Paul/.ssh/id_rsa

SOLUTION 1: Just changing the access rights didn’t worked so I needed to copy it from Windows into the subsystem. After this I could set the access rights and successfully connect to the VM.

$ cp /mnt/c/Users/Paul/.ssh/* ~/.ssh
$ chmod 755 ~/.ssh
$ chmod 600 ~/.ssh/id_rsa

Install nginx

Installing and configuring nginx and Let’s Encrypt (using certbot) is very easy and worked like descripted in all the tutorials. Do not forget to set the DNS record first, before trying to get the certificate.

Now I modified the configuration to redirect one subdomain to the OAuth2 provider and one to the API but it is very important that all request to `/.well-known/acme-challenge` are still directing to the local `wwwroot` folder. This is required for Let’s Encrypt to check if the domains are belonging to you.

The SSL certificate is registered globally for all server configurations. If you like you can drop TLSv1 and TLSv1.1 support as well. But you should never ever enable SSLv3. Due to a bug with nginx and Ubuntu I needed to deactivate gzip compression. It’s not so bad because compression can also be done by the web apps. If everything is setup correct you will now see the empty Azure App page.

3. Deploy the OpenID Connect Provider

Now since the proxy is running I wanted to deploy the OpenID Connect Provider. I have used IdentityServer 4 on top of ASP.NET Core. The source code of my web app is hosted on a private repo at GitLab. If you don’t know GitLab, it’s the only host to my knowledge with unlimited free private repositories and a free integrated CI environment.

Just choose External Repository as deployment option and insert the URL of the repo and the branch. I have a private repository and needed to add the public SSH key to my deployment keys in GitLab; if you have a public repository you can skip this step. You can find the deployment key here: https://mysite.scm.azurewebsites.net/api/sshkey?ensurePublicKey=1

For a more detailed introduction see: Auto-Deploying to Azure App Services from GitLab by Christian Liebel

Problem 2: My Web App didn’t even start ( 502.3 Bad Gateway )

Now, after the app is deployed I have run in the second problem. NOTHING HAPPEND! Exactly I got 502.3 alias Bad Gateway. With Kudu I could identify, that the app is not running and crash immediately after start. Unfortunately, there was no error at all. After sometime I could identify the line of code which crashed:

var cert = new X509Certificate2(certPath, certPass);

SOLUTION: I googled the error and there are 2 steps. First: I needed to upgrade my App Service Plan, because you can’t load certificates under a Shared service plan. So I scaled the plan up to Basic 1. This by itself didn’t solve the problem. Per default the constructor of X509Certificate2 stores the certificate into the user storage which does not exist on Azure. So we need to change the code the following way:

var cert = new X509Certificate2( certPath, certPass, 
X509KeyStorageFlags.MachineKeySet |
X509KeyStorageFlags.PersistKeySet |
X509KeyStorageFlags.Exportable );

Problem 3: Web App can’t connect to SQL Database

No, that the application starts, I couldn’t connect to the database.

SOLUTION was quite easy, just adding the IP of the Authentication App Service to the firewall settings of the SQL Server.

Problem 4: Get the wrong URLs from OpenID discovery request

If you don’t know OpenID Connect, a good way to check if it works is to look for the `/.well-known/openid-configuration`.

You can see in the upper image, the response contained the *.azurewebsite.net URL and not the domain from the reverse proxy. Since the domain is taken from IIS, I could not change it via code and needed to find another way.

SOLUTION: After a little bit of research I found out that I have to specify the following headers in my reverse proxy (/etc/nginx/proxy_param):

And add them to the server config (/etc/nginx/sites-enabled/default):

Problem 5: Error 404 — Web app not found.

Now I got the following error and the proposed solution by MSDN was simple but really difficult to achieve: Just add auth.yourdomain.com to the allowed hostnames. So I tried this but I couldn’t because Azure checks the CNAME / A-Record to look if the domain is actually directing to you App Service. This failed obviously, because the domain is directing to our reverse proxy.

SOLUTION: After a couple of ours searching around the web, I found a note in the old Azure Management Portal:

To verify authorization, create a CNAME resource record with your DNS provider that points from either www.yourdomain.com to my-site-auth.azurewebsites.net, or from awverify.www.yourdomain.com to awverify.my-site-auth.azurewebsites.net.

This was the decisive hint, and so I added the following DNS entry and added auth.yourdomain.com to the allowed hostnames.

awverify.auth 	CNAME	awverify.my-site-auth.azurewebsites.net

As a result, the discovery endpoint now delivers the right URL, but the next error was on the horizon.

Problem 6: OpenID discovery still shows the wrong schema

Although, the right domain is now displayed in the discovery endpoint there is still shown an http as schema instead of https. It’s obviously that the X-Forwarded-Proto header is not applied in the right way. The horrible documented ASP.NET Core doesn’t helped in any way, but after some times searching on GitHub I discovered that I just needed to add the ForwardedHeadersMiddleware to my Startup.cs.

But this didn’t even changes anything. After another couple of hours spent on searching the problem and trying to debug ASP.NET Core apps on Azure (which not really worked well) I found the error and solution.

SOLUTION: Of cause, the X-Forwarded-For header, contained the client IP and the proxy IP, but the X-Forwarded-Proto had only one value: https, the ForwardedHeadersMiddleware saw that as dissymmetry and ignored all X-Forwarded values. And for security reasons, it also ignored every proxy which is not explicitly defined.

After adding this two lines the X-Forwarded headers were processed and everything worked as expected:

to be continued …
One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.