Elasticsearch authentication using Microsoft AAD

Integrating Elastic Cloud deployment with Microsoft AAD

9 min readJul 5, 2023

This blog will focus on integrating Elastic Cloud deployment with Microsoft AAD. This will allow us to control the users' logging into Elastic Cloud deployment's Kibana by getting authenticated via Microsoft's Azure Active Directory (AAD).

Here we will not focus on creating deployment on Elastic Cloud. Please find the official documentation link for the same here.

Now, when we create a deployment on Elastic Cloud, generally, we get two options to log in to get access to our deployment:

Log in with Elasticsearch
Log in with Elastic Cloud

Now, the "Log in with Elastic Cloud" option is typically for users who are administrators and need to have admin privileges on the Elastic Cloud account and all the deployments under it.

However, other users who only need access to the deployment's Kibana have to get their login created with elastic search from the Kibana portal. These users can then log in to the Elastic Cloud deployment using the "Log in with Elasticsearch" option. The users created in the elastic search can be assigned built-in roles (like 'superuser', 'editor', or 'viewer') and custom roles (which can be created and configured under the 'Stack Management' section in the Kibana portal). Though this login option seems simple and does most of the job, it isn't secure in cases where the users might have mistakenly leaked their credentials.

Log in using Microsoft AAD (Azure Active Directory)

We all must have been working with one or the other O365 products in our day-to-day job or might have come across it. These O365 products utilize Azure Active Directory for users to manage authentication and authorization. Integrating our Elastic Cloud deployment's Kibana with Azure AD will allow us to leverage Elastic Cloud deployment with a single identity. Security is another advantage of integrating Elastic Cloud deployment's Kibana with Azure AD.

Users getting authenticated and authorized via Azure AD for Elastic Cloud deployment Kibana login

Steps to integrate Elastic Cloud deployment Kibana with Azure AD:

2. On your “Azure Active Directory” page, select “Enterprise applications” and then click on “New application” to register a new application:

3. Click on “Create your own application”, provide a name (I have taken the name as ‘Elastic Cloud Prod’ here, for example), and select the “Integrate any other application you don’t find in the gallery” option:

This will create our ‘Elastic Cloud Prod’ enterprise application in AAD:

3. Now, before we move ahead with the configuration of this Enterprise application which just registered in our AAD, let's create three different AAD security groups: -

Elastic Cloud Prod Security group (for admins/superusers)
Elastic Cloud Prod Editors (for editors)
Elastic Cloud Prod Viewers (for viewers)

This will help us efficiently manage the users with roles like administrators, editors, and viewers from the AAD security group.

4. Navigate to “Azure Active Directory”, select ‘App registrations’ and search for the name of your enterprise application created in Step 3:

5. Select the ‘Elastic Cloud Prod’ app registration, go to the ‘App roles’ section, and click ‘Create app role’. Provide display name, value, and description, select ‘Users/Groups’ in “Allowed member types”, and enable the app role:

Here we will create three app roles:

Superuser (value: superuser)
Editors (value: editor)
Viewers (value: viewer)

6. Navigate to the new application created, select “Users and groups”, and add all the groups created in Step 3. Assign the role to each group:

7. Navigate to “Single sign-on”:

And edit the basic SAML configuration, add the following information:

a. Identifier (Entity ID) - a string that uniquely identifies a SAML service provider. We recommend using your Kibana or Enterprise Search URL, but you can use any identifier.

For example, https://my-deployment-40837e.kb.us-central1.gcp.cloud.es.io:9243.

b. Reply URL - This is the Kibana URL with /api/security/saml/callback appended.

For example, https://my-deployment-40837e.kb.us-central1.gcp.cloud.es.io:9243/api/security/saml/callback.

c. Logout URL - This is the Kibana URL with /logout appended.

For example, https://my-deployment-40837e.kb.us-central1.gcp.cloud.es.io:9243/logout.

8. Navigate to SAML-based Single sign-on, and open the User Attributes & Claims configuration:

And edit the User Attributes & Claims configuration, select ‘Add a group claim’ and in the ‘Group Claims’ window, select “All Groups” and save changes:

9. Please note the App Federation Metadata Url from the SAML-based Sign-On page:

10. Once all the above steps are followed, then log in to Elastic Cloud with an admin account, and under “Elasticsearch Service” select “Manage” for the required deployment:

11. Under the deployment name, select Edit and then click on Manage user settings and extensions:

a. Update your Elasticsearch user settings with the following configuration:

xpack.security.authc.realms.saml.kibana-realm:
            order: 2
            attributes.principal: nameid
            attributes.groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/groups"
            idp.metadata.path: "https://login.microsoftonline.com/<Tenant ID>/federationmetadata/2007-06/federationmetadata.xml?appid=<Application_ID>"
            idp.entity_id: "https://sts.windows.net/<Tenant_ID>/"
            sp.entity_id: "<Kibana_Endpoint_URL>"
            sp.acs: "<Kibana_Endpoint_URL>/api/security/saml/callback"
            sp.logout: "<Kibana_Endpoint_URL>/logout"

Please update the “<Kibana_Endpoint_URL>” with your deployment Kibana URL, e.g., https://my-deployment-40837e.kb.us-central1.gcp.cloud.es.io:9243. Also, update the “<Tenant_ID>” based on your tenant, available on the tenant overview page in Azure.

In idp.metadata.path paste the App Federation Metadata Url from Step 9.

b. Scroll to the bottom of the configuration page, select Save, and confirm changes:

12. Next, configure Kibana to enable SAML authentication:

a. Follow Step 10, under the deployment name, select Edit, and then click on Edit user settings under Kibana:

b. Update your Kibana user settings with the following configuration:

xpack.security.authc.providers:
  saml.kibana-realm:
    order: 0
    realm: kibana-realm
    description: "Log in with Azure AD"
    icon: "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSAowCPVoJc_t77RCotfTHQixO7KJeV41OxzrKq50vKAux-jVRoY1rzHBMqdLPLeOnATKU&usqp=CAU"
  basic.basic1:
    order: 1

We can update your icon by providing an image URL under the icon key.

c. Scroll to the bottom of the configuration page, select Save, and confirm changes:

Wait for the changes to be applied successfully. Once the changes are applied, open the Elastic Cloud deployment’s Kibana URL in your browser's incognito/in-private window.

You will be able to see three options to log in to Kibana:

As we can see now, the “Log in with Azure AD” option is available with our icon as configured in the Kibana configuration.

However, we can make it more secure by disabling the login option “Log in with Elasticsearch”. To achieve this, follow Step 12 and update your Kibana user settings with the following configuration:

xpack.security.authc.providers:
  saml.kibana-realm:
    order: 0
    realm: kibana-realm
    description: "Log in with Azure AD"
    icon: "https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSAowCPVoJc_t77RCotfTHQixO7KJeV41OxzrKq50vKAux-jVRoY1rzHBMqdLPLeOnATKU&usqp=CAU"

Wait for the changes to be applied successfully. Once the changes are applied, open the Elastic Cloud deployment’s Kibana URL again in your browser’s incognito/in-private window.

Now, you will be able to see only two options to log in to Kibana:

Coming to the last and most important section on how to map our security groups created in Step 3 with Kibana in-built/custom roles.

To achieve this, we must create “Role mappings” in Kibana. This can be done manually and using REST APIs provided by Elasticsearch.

Here, we will focus on REST APIs for our “Role mappings” creation.

Open Azure Active Directory (AAD), go to “Groups” and navigate to three security groups created earlier, i.e., Elastic Cloud Prod Security group, Elastic Cloud Prod Editor, and Elastic Cloud Prod Viewer.

Let’s note down the Object Id of these security groups; we will need it later.

2. Open the Kibana URL and navigate to Menu and look for the Dev Tools option under ‘Management’:

3. Make sure that you are on Console on the Dev Tools window:

4. Enter the below REST APIs for the following role-mappings creation:

i. saml-superuser

#for saml-superuser
POST /_security/role_mapping/saml-superuser
{
"enabled": true,
 "roles": [ "superuser" ], 
 "rules": { "all" : [ 
     { "field": { "realm.name": "kibana-realm" } }, 
     { "field": { "groups": "<Object ID of Elastic Cloud Prod Security group>" } }
 ]},
 "metadata": { "version": 1 }
}

Replace <Object ID of Elastic Cloud Prod Security group> and run the REST API:

ii. saml-editor

#for saml-editor
POST /_security/role_mapping/saml-superuser
{
"enabled": true,
 "roles": [ "superuser" ], 
 "rules": { "all" : [ 
     { "field": { "realm.name": "kibana-realm" } }, 
     { "field": { "groups": "<Object ID of Elastic Cloud Prod Editor>" } }
 ]},
 "metadata": { "version": 1 }
}

Replace <Object ID of Elastic Cloud Prod Editor> and run the REST API:

iii. saml-viewer

#for saml-viewer
POST /_security/role_mapping/saml-superuser
{
"enabled": true,
 "roles": [ "superuser" ], 
 "rules": { "all" : [ 
     { "field": { "realm.name": "kibana-realm" } }, 
     { "field": { "groups": "<Object ID of Elastic Cloud Prod Viewer>" } }
 ]},
 "metadata": { "version": 1 }
}

Replace <Object ID of Elastic Cloud Prod Viewer> and run the REST API:

4. Navigate to Menu and look for the Stack Management option under ‘Management’:

and select Role mappings under ‘Security’, we will be able to see different role mappings created:

Finally, we need to add users to the security group created initially in the Azure Active Directory based on the roles which need to be assigned.

That’s it. Now, you can log in to Elastic Cloud deployment using the “Log in with Azure AD” option on your Kibana login page.

Integrating Elastic Cloud deployment with Microsoft AAD

Log in using Microsoft AAD (Azure Active Directory)

References:

Written by Gaurav Kumar