Connect Azure Kubernetes Java Applications to SQL with Kerberos Integrated Authentication

In some cloud migration journeys SQL server database moves are done using lift and shift approach. Azure provides option to host SQL server on VM to enable fast migration. The challenge we recently encountered is that SQL server on VM has only two options for authentication: SQL Authentication (aka username/password) and Windows Authentication (facilitated with Kerberos tokens). In this article we will demonstrate how to connect containerized Java Applications running on AKS Linux nodes with SQL server using Kerberos authentication. The full application and setup could be found on https://github.com/lenisha/jdbc-kerberos

Table of Contents

Architecture
Preparing Azure Infrastructure
Kerberos setup
- Create Keytab File
- Create Kerberos Config krb5.config
Java Application Code and Container
Token Refresh SideCar Container
Kubernetes Setup
- Store KeyTab in AKV Secret and map with K8S AKV CSI Volume
- ConfigMap for Kerberos Config krb5.conf and SQLsettings
- Kubernetes Application POD manifest
Verify Kubernetes Application
Alternative Setup Options
Conclusion

Architecture

Here is the diagram describing architecture enabling Java container applications to connect to SQL server.

• Managed Domain on Azure Active Directory Domain Services acting as KDC (Key Distribution Center) and DNS
• SQL server on Windows VM domain joined with AADS Managed Domain
• Kerberos encrypted keys (Keytab file) stored as secret in Azure KeyVault
• Azure Kubernetes Cluster (AKS) with Linux node pool and enabled Managed identity and KeyVault CSI Driver to retrieve secrets
• Java application with JDBC driver settings utilizing Kerberos Authentication
• Sidecar container refreshing Kerberos tokens used by JDBC driver

Preparing Azure Infrastructure

Prepare two virtual networks one for AADS and one for AKS/SQL/KeyVault and peer them as described in VNET peering to enable connectivity.

Create and configure an Azure Active Directory Domain Services (AADS) managed domain as per tutorial: https://docs.microsoft.com/en-us/azure/active-directory-domain-services/tutorial-create-instance and update VNETs DNS settings to point to AD Domain Service DNS IPs. (Note: while we have chosen AADS as simple and managed solution, you could deploy Windows Datacenter VM to act as Domain Controller/KDC)

ADDS Managed domain

Create SQL Server on Azure Windows VM and join SQL VM with AADS managed domain.

Infrastructure setup for AADS managed domain and SQL

Create DBUser in Azure Active Directory ( it will be synced with AADS)

Add DB user to SQL Server Database and grant permissions. Connect to SQL VM using Bastion and use SSMS (SQL Server Management studio) to create a “testdb” and enable DOMAIN\dbuser (DOMAIN is name of AADS managed domain):

CREATE LOGIN [ENEROSORG\dbuser] FROM WINDOWS
CREATE USER [ENEROSORG\dbuser] FOR LOGIN [ENEROSORG\dbuser];  
ALTER ROLE db_owner ADD MEMBER [ENEROSORG\dbuser];

Create Azure KeyVault (with private endpoint) and Azure Kubernetes Cluster in the peered VNET.

Kerberos setup — create Keytab

For application to perform Kerberos authentication we need to create Keytab file (MIT Keytab definition) with dbuser principal and encrypted keys that will be used for acquiring Kerberos tokens from KDC.

To make keytab use MIT utilities for Linux. Provision Ubuntu Linux VM (no need to join the domain) in the same VNet as SQL VM and install Kerberos utilities:

sudo apt-get update
sudo apt-get install krb5-user samba sssd sssd-tools libnss-sss libpam-sss ntp ntpdate realmd adcli

while installing the packages you will be prompted to provide default KDC realm name AADS managed domain in our case, e.g ENEROSORG.ONMICROSOFT.COM (should have all CAPS) and default /etc/krb5.conf Kerberos configuration file will be created.

Use ktutil utility to add user principal encrypted keys and save keytab to the file:

> ktutil
ktutil:  addent -password -p dbuser@ENEROSORG.ONMICROSOFT.COM -k 2 -e aes128-cts-hmac-sha1-96
ktutil: wkt dbuser.keytab

Verify and use Keytab file to initialize the token

> sudo kinit -V -kt dbuser.keytab  dbuser
Using default cache: /tmp/krb5cc_1000
Using principal: dbuser@ENEROSORG.ONMICROSOFT.COM
Using keytab: dbuser.keytab
Authenticated to Kerberos v5

Test Kerberos token to establish connection to SQL server using sqlcmd (Install sqlcmd)

> sqlcmd -E -S <SQLServer>.ENEROSORG.ONMICROSOFT.COM -d testdb -Q "SELECT SUSER_SNAME();"
-------------------------------------------------------------------
ENEROSORG\dbuser

Kerberos setup — create krb5.config

While in the steps above we used default Kerberos configuration created during client setup, we will customize configuration file (similar to JDBC driver example) to use our managed domain name as default realm and set location for default keytab file used for token generation and type/location for token cache ( same path as KRBCCNAME env variable in Java dockerfile)

[libdefaults]
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true
dns_lookup_realm = false  
ticket_lifetime = 24h  
renew_lifetime = 7d
forwardable = true
rdns = false
default_realm = ENEROSORG.ONMICROSOFT.COM
default_keytab_name = /krb5/dbuser.keytab
default_ccache_name = FILE:/dev/shm/ccache
default_tkt_enctypes =  aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5
default_tgs_enctypes =  aes128-cts-hmac-sha1-96 arcfour-hmac-md5 des-cbc-crc des-cbc-md5
[realms]
ENEROSORG.ONMICROSOFT.COM = 
{  
  default_domain = ENEROSORG.ONMICROSOFT.COM  
}
[domain_realm]
.enerosorg.onmicrosoft.com = ENEROSORG.ONMICROSOFT.COM
enerosorg.onmicrosoft.com = ENEROSORG.ONMICROSOFT.COM 
[logging]
default = STDERR

Very important is to include encryption algorithm that was used for keytab generation in our default settings as shown above.

Java Application

JDBC driver docs provide a great summary of the process to setup Kerberos authentication Using Kerberos integrated authentication to connect to SQL Server. To be able to run on Linux containers Java app will use pure Java Kerberos implementation (instead of OS native dll for Windows) Krb5LoginModule. You must specify the integratedSecurity=true and authenticationScheme=JavaKerberos connection properties as shown in lines 20 and 21 below:

This Java application connects to database and continuously prints authenticated user. For JDBC driver to be able to get Kerberos tokens from KDC following steps are required:

1. Install Kerberos client packages
2. Set the default Kerberos ticket location. This is done by setting the KRB5CCNAME environment variable.
3. Setup Kerberos domain configuration file for the realm. E.g. https://github.com/lenisha/jdbc-kerberos/blob/main/containers/krb5.conf
4. Get the Kerberos ticket, by generating one using keytab file and kerberos utility: kinit -V -kt USER.keytab USERwhere "USER" and "DOMAIN.AD" is the principal and domain respectively. The ticket will be generated in the default ticket location or in the KRB5CCNAME path if set.

Java Application Container Configuration

To complete all these steps for Java Application container, following Dockerfile is used

FROM maven:3.8.4-jdk-11-slim as maven 
COPY ./pom.xml ./pom.xml
COPY ./src ./src
RUN mvn clean compile assembly:single 
FROM openjdk:11-jre-bullseye 
WORKDIR /app
COPY --from=maven target/sql-kerberos-jar-with-dependencies.jar /app/sql-kerberos.jar 
ENV DEBIAN_FRONTEND=noninteractive 
RUN apt-get update
RUN apt-get install -y krb5-config
RUN apt-get install -y krb5-user
RUN mkdir /krb5 && chmod 755 /krb5 
VOLUME ["/krb5","/dev/shm","/etc/krb5.conf.d"] 
ENV KRB5CCNAME=/dev/shm/ccache
CMD ["java", "-jar", "sql-kerberos.jar"]

Container image uses multi-stage build to compile and package Java application and prepare runtime image with required Kerberos client utilities, token cache location variable and volumes that would contain Kerberos Domain configuration and Keytab. No environment or keys are stored inside the image, they will be mapped to container using Kubernetes resources.

Token Refresh SideCar Container

For Java application to be able to use Kerberos token it should be generated and periodically refreshed. We will use Sidecar container approach described in Kerberos Sidecar Container Blog . Sidecar will use kinitutility to generate tokens and shared memory volume to share them with Java container. Script and dockerfile we use could be found in https://github.com/lenisha/jdbc-kerberos/blob/main/containers/sidecar .

Kubernetes Setup

KeyTab Secret AKV CSI Volume

Now we could load dbuser.keytab file into Azure KeyVault secret preserving its binary content by setting encoding to hex

> az keyvault secret set --name dbuserkt --vault-name <vault name> --file dbuser.keytab --encoding hex

Enable AKS addon Azure Key Vault Provider for Secrets Store CSI Driver it will create Managed identity named azurekeyvaultsecretsprovider-*that we could use to access KeyVault. (in production AAD Pod identity should be used as opposed to cluster wide identity) Grant KeyVault secret access to addon’s Managed Identity.

Create SecretProviderClass that would define access to KeyVault secret and map it to file in the Pod’s Volume (https://github.com/lenisha/jdbc-kerberos/blob/main/containers/kvprovider.yml)

Krb5.conf and SQL ConfigMap

Create ConfigMap with krb5.configfile contents and ConfigMap with database location settings:

kubectl create configmap krb5config --from-file=./containers/krb5.conf
kubectl create configmap dbconfig --from-literal=SQL-SERVER=SQLSERVER.ENEROSORG.ONMICROSOFT.COM --from-literal=DB-NAME=testdb

Kubernetes Application POD

To provide all the required settings to the containerized application use Kubernetes resources that allow to provide settings and secrets as files mounted to the Pod’s volumes.

To setup application as per the picture above apply following manifest:

where

Line 7–23: defines Sidecar container for token refresh

Line 25–39: defines Java application container

Line 42–43: specifies Shared memory Volume for sharing Kerberos token between containers and it is mounted to /dev/shm path to both containers Line 16–17 and Line 32–33

Line 44–49: Specifies Volume that is mapped from ConfigMap containing Kerberos configuration and mounted to both containers on Line 18–19 and Line 34–35 to path /etc/krb5.conf expected by kerberos client tools installed in containers

Line 50 -55: Specifies AKV CSI Driver Volume that maps AKV secret containing user Keytab File and is mounted to both containers Line 21–23 and Line 37–39 to /krb5/dbuser.keytab path. This is the path that is set in Kerberos configuration as default_keytab_name

Verify Kubernetes Application

Once Kubernetes Manifest is applied , verify that both containers are running in the Pod:

Verify Sidecar Logs by looking at the logs for kinit container in the pod , you will see output from refresh token script and it will list tokens available in the cache:

Sidecar Logs

Verify successful connection to DB from Java application , it would print principal for the connected user:

Java App Logs

Alternative Setup Options

In this article we have used few infrastructure choices, here are few alternatives:

• Azure AD Domain Services (AADS) could be replaced with your own Domain Controller on Windows VM deployment
• Instead of managing your own SQL server, use Arc enabled SQL MI and benefit from reusing pre-build automation and scale.

To setup this infrastructure please refer to excellent article by colleague of mine Raki Rahman https://www.rakirahman.me/arc-sqlmi-active-directory/

Conclusion

This article demonstrated end to end setup for containerized Java Application running on Kubernetes Linux Nodes and connecting to SQL Server on Azure VM using integrated Kerberos Authentication with Azure AD Domain Services acting as KDC. We showed how to create Keytab to enable Kerberos token generation and store it securely in Azure KeyVault.

This setup will be greatly simplified once SQL Server on Azure VM will support AzureAD authentication.