How to enable Windows Authentication for Load Balanced CyberArk AAM API calls
I figured I would write this blog related to supporting CyberArk’s Application Access Manager (AAM), (aka Application Identity Manager (AIM), aka Central Credential Provider (CCP)), and configuring Windows Authentication with a load balancer IF the load balancer is modifying traffic. Yes; they won’t stop renaming the offering… I still mostly call it CCP. Sorry.
CyberArk’s documentation and support around configuring CCP for Windows Authentication is mostly lacking as they punt to Microsoft and/or your load balancer on how to configure CCP with OS Authentication.
Why should you care?
- If you have CyberArk CCP deployed in your environment with Load Balancers modifying header packets, such as adding x-forwarded-for.
- To allow for more secure authentication and authorization of CCP API call usage for passwords/secrets retrieval from your CyberArk Vault.
- We need to make it harder on attackers. The bare minimum for CCP API calls is allow-listing IPs, or a range of IP addresses.
Is this a silver bullet to solving IT Security?
NOPE. Google ”Kerberos attacks” after you read through this.
CyberArk what, and a what?
CyberArk’s Vault is the secure repository of all sensitive information, and it is responsible for securing this information, managing and controlling all access to this information, and maintaining and providing tamper-proof audit records. Translated; this means your sensitive passwords, secrets, ssh keys, etc., will be secured within the digital vault.
CyberArk’s CCP is designed to provide comprehensive privileged access, credential, and secrets management for widely used application types and non-human identities. Translated; this means you can perform CCP API calls with CyberArk’s Vault to securely pull credentials. This is one offering of CyberArk’s which allows you to remove hard-coded passwords within config files, or scripts.
Note:
Each environment is different. What I’m describing below should be performed in your own lab environment before deploying to production. You’re responsible for your own actions and validating the pieces required that may be different in your environment. There are a variety of additional security settings which you can and likely should configure in addition to these items.
Why doesn’t Windows Authentication work automatically through a VIP?
- In short, due to a Kerberos double-hop issue. In this context, we need to be able to pass on a user’s Kerberos token on in order to authenticate the user and/or service account performing the CCP API call(s). The CCP API call will hit the VIP, then be passed onto a server, which breaks this. This normally is because when your VIP load balancer is configured to modify traffic passing through.
- Kerberos double-hop issues are described further here: https://techcommunity.microsoft.com/t5/ask-the-directory-services-team/understanding-kerberos-double-hop/ba-p/395463#:~:text=Kerberos%20Double%20Hop%20is%20a,further%20connections%20to%20other%20servers .
What is unconstrained and constrained delegation?
Unconstrained delegation — BAD. Don’t do this. Allows a service account to delegate (impersonate) Kerberos access for every account in the domain. https://www.cyberark.com/resources/threat-research-blog/weakness-within-kerberos-delegation
Constrained delegation: Better. This limits privilege to only what is needed for the functionality you’re looking for. This, plus adding privileged accounts to “protected users” security groups, makes it so service accounts can’t impersonate any sort of privileged accounts you may have in your environment.
https://docs.microsoft.com/en-us/windows-server/security/kerberos/kerberos-constrained-delegation-overview
Reminder:
If your load balancer is NOT modifying traffic, a LOT of these steps are NOT required. For example, constrained delegation, a service account SPN, setting a svc account for the app pool. The traffic will simply pass through after you configure IIS correctly using Windows Authentication.
What you need:
- Active Directory (AD) — standard for an enterprise
- 1 service account (domain user) — For example, domain\svc_auth
- Constrained Delegation on the service account — domain\svc_auth
- Create and Set Service Principal Name (SPN) for domain\svc_auth service account - CyberArk Vault
- 2 Windows servers, joined to your Active Directory domain;
Running IIS, with CyberArk CCP installed - Load balanced VIP using whatever technology (1 global & 2 separate local VIPs which normally are in different data centers)
- Signed Certificates loaded for secure HTTPS traffic to the VIPs tied to the IIS sites.
- 1 client (server or workstation), on the same domain
- Configured Safe with necessary CP Provider Permissions & AppID permissions. When configuring the AppID’s OS Authentication, the windows domain prefix must be included prior to the username. For example, domain\username .
Make it so:
We need to configure multiple items for this to all work, similar to what’s described here*.
- Load balancer is configured to forward on CCP API requests coming into VIP URL. The VIPs and members are enabled for traffic, and have X-Forwarded-For headers included so we can determine the client performing the API calls to the VIP.
- On the CCP IIS websites, update the application pools to utilize “custom identity”. In this case, a limited (NOT ADMIN) service account called domain\svc_auth . You’ll need the current password for the service account to set this.
- On the AimWebservice sub-folder, set Windows authentication to enabled. Every other authentication set to disabled. Set Providers to Negotiate:Kerberos & Negotiate. Set useAppPoolCredentials=True. For advanced settings, set Extended Protection to Accept. Set Kernel-mode authentication to enabled.
- Depending on your CCP IIS folder location and deployment, you may need to set the top-level to anonymous authentication to allow users to reach the CCP site. You have to understand your environment on how best to configure this part. DON’T DO this across the board for your whole site. Bad. - On both IIS Servers, set an OS server configuration (Local Security Policy> User Rights Management) — Allow domain\svc_auth service account to logon as a service.
- Configure and enable CONSTRAINED delegation for domain\svc_auth. In AD under the domain\svc_auth properties\Delegation tab, Set to “trust this user for delegation to specified services only”> “Use Kerberos only”. Note that the VIP name(s) you have configured will not show up in Active Directory for selection and use.
- HTTP/ccpServer1
- HTTP/ccpServer1.domain.com
- HTTP/ccpServer2
- HTTP/ccpServer2.domain.com
7. Configure SPN items for Kerberos authentication for domain\svc_auth for the VIP member servers and VIP. This includes both the short name and Full Qualified Domain Names (FQDN) configured for the 2 servers. If desired, you can include a specific port for IIS and these as well. Update to fit your needs.
- setspn -a HTTP/ccp domain\svc_auth
- setspn -a HTTP/ccp.domain.com domain\svc_auth
- setspn -a HTTP/ccpServer1 domain\svc_auth
- setspn -a HTTP/ccpServer1.domain.com domain\svc_auth
- setspn -a HTTP/ccpServer2 domain\svc_auth
- setspn -a HTTP/ccpServer2.domain.com domain\svc_auth
8. Perform IISRESET on both servers after configurations are finished.
9. Restart both CCP IIS servers.
10. Wait until both CCP IIS servers are back up.
OK, Lets Ride!
Here’s a Powershell one-liner for CCP API Call validation:
Invoke-RestMethod -Uri “https://ccp.domain.com/AIMWebService/api/Accounts?AppID=CCP-AppIDHere&Safe=SafeNameHere&UserName=TestAccountHere&Reason=CCP Authentication Test” -UseDefaultCredentials
YESSSSSS…
Successful CCP API Call result:
- Content: <PasswordContentHere>
- CreationMethod: PVWA
- Address: <Addresshere>
- Safe: <SafeNameHere>
- UserName: <TestAccountHere>
Failed CCP API Call result:
- Invoke-RestMethod : {“ErrorCode”:”APPAP306E”,”ErrorMsg”:”Authentication error for App ID [Database]. Reason: APPAP133E Failed to verify application authentication data: OSUser \” \” is unauthorized”}
https://cyberark-customers.force.com/s/article/CCP-AIMWebService-Windows-Authentication-not-working
DUDE. It isn’t working:
- It took me a good amount of time to make all these these parts function as expected. Check here for troubleshooting ideas: https://docs.microsoft.com/en-us/troubleshoot/developer/webapps/iis/www-authentication-authorization/troubleshoot-kerberos-failures-ie
- Looking to buy plugins pre-packaged? Check out my site here — https://www.keyvaultsolutions.com/
- Message me on Twitter Joe Agler if you have any questions, issues, etc.
Additional items to consider:
- Vulnerability scan your site(s) afterwards. Ensure you resolve any necessary findings. Don’t YOLO this, especially if your site is publicly available.
- Password management of the domain\svc_auth account and IIS usages
- Add any additional security configurations for your environment, ideally via Group Policy.
- Consider adding authorization rules for certain AD Security groups for IIS— https://docs.microsoft.com/en-us/iis/configuration/system.webserver/management/authorization/authorizationrules/scope/add#:~:text=On%20the%20IIS%20Manager%20Permissions,User%20dialog%20box%2C%20click%20OK. This does add an additional layer of complexity however.
- Validate that Kerberos authentication is functioning as expected. Meaning, it’s not utilizing NTLM.
- Add your Active Directory domain privileged accounts to “protected users” security groups.
K,THX,BYE.
More Reading if you want:
*Kerberos Double Hop Delegation from IIS to SQL server (using django) — https://serverfault.com/questions/928911/kerberos-double-hop-delegation-from-iis-to-sql-server-using-django
https://adsecurity.org/ — Tons of IT Security recommendations, primarily related to Active Directory.
