Journey to Automated Permissions Management
One of the hardest things to do within any cloud environment is to manage permissions for developers and business people, enabling them to do their jobs whilst still keeping infrastructure and customer data secure. Do you follow the principle of least privilege and give them the minimum access possible, then deal with the inevitable flood of support requests when greater privileges are required? Or, do you give them more access, with the security risks that entails?
This is the dilemma we faced at Perkbox. We had up to 10 requests for temporary access every day when a lot of testing or prototyping work was happening. In most cases we would need to ask multiple questions to the requester to find all the details, resources, time limits etc. A lot of the time the answer was just to use infrastructure as code (We make pretty extensive use of Terraform). Not every task can be accomplished through IAC though (for example when exploring a new AWS tool, or when temporary database access is required). All the questions could lead to 20 minutes of back and forth until we got it right, slowing everyone down and making rapid development a lot harder.
Finding a solution
We were not willing to compromise our security, so permanently elevating people’s permissions to reduce the support burden was not an option. But we needed to reduce the amount of time managing people’s access took up day to day. We came up with the following criteria that any solution to this problem needed to meet:
- Be able to grant temporary, limited in scope and time limited access to individual users
- Be simple to use
- Provide an audit trail
- Ideally work within existing tools (e.g. slack)
We looked around at what tooling was currently available around this, as well as coming up with a few ideas of our own, eventually narrowing down the options to:
- A CLI tool to allow developers to generate the IAM policy to post in a support request
- A open source tool from Netflix called Console Me
- A Slack Bot to contain the whole request and approval process
Idea 1: CLI Tool
Our first choice was a CLI tool. In the Devops team we distribute a tool to developers to help improve their interaction with the platform (e.g. simplifying access to various systems). It seemed like a perfect fit for a new permission request system.
The tool was developed over a few weeks. It allowed developers to select 90% of the details and automatically generate an AWS IAM policy which could be added to their support requests to speed up the grant of temporary permissions.
This solution still had a few shortcomings:
- Auditing still involved manually creating Jira tickets from the slack support messages.
- Copy/paste issues from the terminal, which broke the policies in some cases.
- No easy way to ensure that all the fields were correctly populated.
Idea 2: Netflix ConsoleMe
While researching existing tooling, we came across Netflix’s ConsoleMe, which is a tool for managing temporary AWS IAM permissions in a Web based GUI (it also provides a CLI tool).
On the surface this sounded like it would provide what we were looking for, so why didn’t we go with it?
ConsoleMe has a few drawbacks for us:
- Not a simple tool to set up/complex to maintain. We are a small team, and wanted something that would reduce our support burden, not more complex tooling to maintain. ConsoleMe is probably better suited for larger organisations.
- The user interface is fairly complicated. It would have required some developer training in order for them to use it effectively.
- It wouldn’t integrate with our existing tooling. Developers already authenticated to AWS via their company Google accounts, leveraging AWS IAM’s built-in support for identity providers and federation (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html). ConsoleMe doesn’t support this. Adopting it would have meant changing this setup and requiring AWS users to use a different GUI (ConsoleMe) or CLI tool (Weep) just to access AWS.
If you are a medium to large organisation and don’t have any existing auth integrations in place then ConsoleMe might be a good solution, but it wasn’t a good fit for our needs.
Idea 3: Slack Bot
The slack bot idea had been initially skipped as none of the engineers had any prior experience of building a Slack bot. Though after making some progress with our first attempt (the CLI tool) we decided this was the best way forward.
Whilst doing some research we came across this medium post, which helped with getting started -
Implement Slack Slash Command in Golang using Socket Mode
This provided us with a simple bit of code to get a basic Slack slash command bot up and running. One of the key things with a Slack bot which we all liked was it was a single place for the generation, approval and even auditing at a high level. Slack’s UI isn’t all that bad looking either, and the modals are fully customisable.
The Slack bot was our chosen solution in the end. It covered almost everything we needed (auditing, simple to use, a nice UI and input validation). This gave birth to the Cloud Access Bot, a simple easy way for developers to request temporary extra permissions to AWS accounts.
The flow of the bot is:
- User enters slash command `/request`
- Slack presents Modal requesting details for access (Account, Role, Time, Resources, Actions) Note: The Bot currently doesn’t allow for wildcard resources or permissions.
- Model submitted and Request is sent to Approvers
- Once approved the Role is Applied
- User is notified that the extra permissions have been granted.
The revoking of the permissions is nicely handled by IAM’s own `DateFrom` and `DateTo` conditions on the policies.
For Admins (the DevOps team in our case) the approval process is pretty simple. Just click the “Approve” or “Deny” button on the request after a short review of the requested permissions.
If the request is approved by the admin, the bot will apply the permission policy to the selected role, and then alert the requester to the permissions were granted.
If the request is denied by the admin, then the requester will be notified that the request was denied, and no policy will be applied.
Who is considered an approver is fully customisable based on Slack user groups.
The inline policy applied for the request above is:
We ran into a few challenges during development, to mention a few:
- As previously mentioned, no real development experience across the team on how to write a Slack bot
- Something that we also ran into with the CLI tool. There are no good libraries out there (either by AWS or third parties) that help with generating AWS policies. Generating ARNs for different resource types is a challenge in particular, as AWS don’t publish any schema for this. There are libraries out there that have attempted to scrape the info needed from AWS IAM reference docs, but the documentation is inconsistent.
- Slack’s API presented a few challenges, for example, static option lists have a limit of how many items can be displayed (100 Options), with a character limit (2000 characters), making it tricky to present long lists of resources for selection.
End Result
Through some experimentation and trial and error (mostly related to the Slack API), we finally arrived at a usable Slack bot. The end result is we now have a quick and simple process for developers to request and be given time limited privileges with an audit trail.
So why are we talking about this tool in the first place, other than it being a pretty interesting project?
Well, we open sourced it!
https://github.com/perkbox/cloud-access-bot
In the project you will find detailed instructions on how to get set up, what the current limitations are, and how to make your own contributions (should you so wish).
Enjoy!
