awless.io, a mighty CLI for AWS

Some people strongly prefer the CLI (source: Imgur)

As the cloud has grown in popularity, AWS has become the undisputed market leader, powering many of the internet giants and setting the pace for the wide range of cloud services. The challenge for DevOps is to keep up with that pace so they can understand the growing number of services and their associated features.

DevOps often wonder about the best ways to manage quick-growing AWS infrastructures. They question whether they should use the graphical user interface (GUI, a.k.a. the AWS console) or the command line interface (CLI, starting with AWS’s own aws-cli)? Or maybe a mix of both?

There is another option to proposes to simplify the work of DevOps : awless.io, a new open source CLI for AWS written in Go. The awless.io project got praise from AWS superstar Jeff Barr and attracted more than 2.5k stars on GitHub in just a few months. In this blog, I’ll explain why my company Wallix built awless.io and why we think the tool can help DevOps, as many tweets suggest.

A Love/Hate Relationship with GUIs

AWS provides many services

Usually, sysadmins hate GUIs. The Linux Foundation considers them for novices. But could AWS Console be the exception? The Console clearly presents the full range of available services, and, depending on the service, provides advanced wizards to help manage the features of a given service. We do appreciate the Console, especially for infrequent actions because the graphical visualization is very helpful. But repeating tasks takes time and can easily frustrate advanced users.

Even at Step 6 using AWS Console, we haven’t created our instance yet.

With the GUI, it is hard to script actions, keep notes for oneself, or collaborate with a colleague about a particular operation. Remember the last time you instructed a colleague (or your mother) over the phone while navigating web interfaces? Needless to say, it’s a very frustrating process.

The Burden of the aws CLI

A CLI, on the other hand, is fantastic for all actions that are performed regularly. Once the aws CLI is setup, it can perform any action without requiring you to login manually or use a mouse.

For example, this command suffices to launch a t1.micro instance:

> aws ec2 run-instances --image-id ami-xxxxxxxx --count 1 --instance-type t1.micro --key-name keypair-xxxxxxxx --security-groups sg-xxxxxxxx

We could argue that it’s hard to remember the Amazon Machine Image (AMI) identifier or the keypair, but a shell alias would do the job. We could simply search for the right security group:

> aws ec2 describe-security-groups --group-name "awless-secgroup"

and grep the GroupId from the input, which looks by default like:

While this approach feels much more sophisticated than using AWS Console, it is not always efficient. The aws CLI requires a constant switch from input: Series of command lines are intertwined with extracting information from previous output or editing snippets of JSON. Some open source alternatives exist, but they barely change the syntax of operations, adding coloring, completions and other niceties.

Tools, such as AWS’s own CloudFormation, are helpful for preparing a stack. But writing Templates is a separate task that can’t replace many simple commands. We once witnessed AWS experts struggling to find resources created by a CloudFormation template using the aws CLI. Grokking the right id in the middle of kilobytes of JSON is not an easy task. In the end, we had to resort to using the GUI.

Enter awless.io

awless.io is an alternative CLI for AWS that values simplicity over exhaustivity. It aims to perform 90 percent of tasks much more easily by changing the definitions of commands.

awless.io is not just a “front-end” to aws-cli. It is a new CLI implemented in Go using the official AWS Go SDK. The project is fully open source project and released under the Apache license. As of now, awless.io supports the following AWS services: IAM, EC2 (including ELBv2), S3, Route53, RDS, SQS, SNS, Cloudfront, Cloudwatch, CloudFormation and Lambda. Even more services will be added in the near future.

The awless.io project was started by the Innovation team at my company, Wallix, stemming from our own needs. Wallix is the de facto sponsor for the project, but ultimately awless.io is a community project and welcome all contributions.

The Benefits of awless.io

awless.io is modeled after popular tools, such as git, and most commands are in the form of:

awless [verb] [noun] [parameter=value ...]

For exemple, the following are all valid commands:

# List all existing users
awless list users
# Create a new instance
awless create instance
# Attach a user to a group
awless attach user name=my-user-name group=my-group
# Run a template
awless run https://example.com/create_vpc.aws

The main principles behind awless.io include:

  • Smart defaults to minimize cloud knowledge;
  • Output that is readable by humans or parsable by machines;
  • Local copy of the infrastructure model is stored as a RDF graph, which enables users to study the graph and perform computations locally.

awless.io highlights the reuse of the same verbs (such as list, create, delete), which results in a small (and easy to remember) command surface. The syntax is unified across all services. For example, you can type:

awless list [instances|users|buckets|records]

instead of

aws ec2 describe-instances
aws iam list-users
aws s3api list-buckets
aws route53 list-resource-record-sets

Getting Started with awless.io

awless.io is available at GitHub. We provide pre-built binaries and Homebrew packages for macOS:

brew tap wallix/awless; brew install awless

Alternatively, building this yourself is very easy if you have Go installed on your system:

go get -u github.com/wallix/awless

At first launch, awless.io automatically gets your credentials and default region if the official aws-cli is setup. Also, don’t forget to install the autocompletion for bash or zsh.

Exploring AWS Infrastructure

awless.io provides many features and refinements to help DevOps navigate through their AWS infrastructure. The two main commands are list and show. For each type of resources, awless.io tries to provide meaningful information first.

Listing Objects

awless.io enables to list all existing AWS resources, like users, security groups or instances:

> awless list users
| ID ▲ | NAME | LASTUSED | CREATED |
|-----------------------|-----------|----------|-----------|
| AIDAI37GETCRRVNLDKFYQ | hbinsztok | 8 days | 10 months |
| AIDAI3FX74KAQQKIOM2XY | awless-ts | 13 weeks | 4 months |
[…]

When the terminal is not wide enough, columns are truncated automatically and a warning message indicates the hidden columns:

> awless list securitygroups
| ID ▲ | VPC | INBOUND |
|-------------|--------------|-------------------------------------|
| sg-1d578b64 | vpc-00b68c65 | [0.0.0.0/0](tcp:1433) |
| | | [0.0.0.0/0;::/0](tcp:443) |
| | | [::/0;0.0.0.0/0](tcp:80) |
| | | [::/0;0.0.0.0/0](tcp:2242) |
| | | [::/0;0.0.0.0/0](tcp:22) |
| | | [0.0.0.0/0](tcp:3389) |
| sg-30abc756 | vpc-00b68c65 | [0.0.0.0/0](tcp:443) |
| | | [0.0.0.0/0](tcp:80) |
[...]
Columns truncated to fit terminal: 'Outbound', 'Name', 'Description'

You can sort and filter content easily:

> awless list instance --sort uptime --filter zone=eu-west-1b
| ID | ZONE | NAME | STATE |
|---------------------|------------|---------------------|---------|
| i-03a1f22b36ddb0739 | eu-west-1b | blog-test | running |
| i-05adde5a930da5ed3 | eu-west-1b | authoringtest | stopped |
| i-06b8f07c1e4afb49a | eu-west-1b | AwlessWithScheduler | running |
[...]
Columns truncated to fit terminal: 'Type', 'Public IP', 'Private IP', 'Uptime', 'KeyPair'

Output format is customizable. By default, the output appears in tables that are easily read by humans (the output is a valid markdown, and can be piped to other tools, such as pandoc). You can choose instead csv, json or tsv:

> awless list subnets --format csv
ID,Name,CIDR,Zone,Default,Vpc,Public,State
subnet-0c41ad68,,172.31.0.0/20,eu-west-1a,true,vpc-00b68c65,true,available
subnet-2486b17c,demo-env-subnet,10.0.0.0/24,eu-west-1c,false,vpc-67b25c00,true,available
[...]

A distinctive feature of awless.io is the ability to work from a local copy of the infrastructure graph, stored as a RDF graph. This approach enables you to perform complex queries locally. Also, the use of a standard format enables you to reuse existing tools to query the graph, such as Protégé, which was developed at Stanford. Having a local graph enables to easily query your infrastructure even without internet access:

> ping amazon.com
ping: cannot resolve amazon.com: Unknown host
> awless list instances --local
| ID ▲ | ZONE | NAME | STATE |
|---------------------|------------|---------------------|---------|
| i-01b17116fed0a09bd | eu-west-1c | windows-demo-env | running |
| i-05c0a275adb5484fa | eu-west-1c | git.inno.wallix.com | running |
| i-06b8f07c1e4afb49a | eu-west-1b | AwlessWithScheduler | running |
[...]

Getting Resources Details

You can get the details of any kind of resource using the show command:

> awless show i-03a1f22b36ddb0739
| PROPERTY ▲ | VALUE |
|-------------------|---------------------|
| Architecture | x86_64 |
| Hypervisor | xen |
| ID | i-03a1f22b36ddb0739 |
| Image | ami-70edb016 |
| KeyPair | keypair-blog |
| Name | blog-test |
| NetworkInterfaces | [eni-10e4352d] |
| Private IP | 10.0.20.245 |
| Public IP | 34.252.193.183 |
| RootDevice | /dev/xvda |
| RootDeviceType | ebs |
| SecurityGroups | [sg-a2a282c4] |
| State | running |
| Subnet | subnet-472a1731 |
| Type | t2.micro |
| Uptime | 46 hours |
| Vpc | vpc-96c7eef2 |
| Zone | eu-west-1b |
# Relations:
eu-west-1[region]
↳ @awless-test-vpc[vpc]
↳ @awless-public-subnet[subnet]
↳ @blog-test[instance]
Depending on: keypair-blog[keypair], @default[securitygroup], vol-0f82cd93824f456e5[volume]
Siblings: @AwlessWithScheduler[instance]

This command will display the properties of the cloud resource, as well as its relations with other resources. For example, for an instance, it will show its Virtual Private Cloud (VPC), subnet and related objects, information about mounted volume, keypair and its security group(s).

Properties and values will differ depending on the type of AWS resource:

> awless show AIDAI3FX74KAQQKIOM2XY
| PROPERTY ▲ | VALUE |
|------------------|-----------------------------------------------|
| Arn | arn:aws:iam::519101999238:user/awless-test-usr|
| Created | 4 months |
| ID | AIDAI3FX74KAQQKIOM2XY |
| Name | awless-test-user |
| PasswordLastUsed | 2 weeks |
| Path | / |
Depending on: @PepsWallixS3Assume[policy], @PepsWallixS3[policy]

One-Liners and Templates

awless.io is a great tool for listing and exploring your cloud resources. But it also provides a powerful templating engine, which supports the cloud infrastructure operations (creation, update or deletion of resources).

One-Liners

awless.io templates can either be used through one-liner shortcut commands or with file scripts. To create an instance or a keypair or any other entity, you only need to type a simple command:

> awless create instance
> awless create keypair

awless.io does not require any additional parameters right away to create an instance or perform other actions. Instead awless.io will input for the missing parameters and provide autocompletion along the way.

Let’s first create a keypair to enable to further connect via SSH to instances we will create next (user input is in bold):

> awless create keypair
Please specify (Ctrl+C to quit, Tab for completion):
keypair.name? keypair-blog
create keypair name=keypair-blog
Confirm? (y/n): y
[info] Generating locally a RSA 4096 bits keypair…
[info] 4096 RSA keypair generated locally and stored in ‘/Users/henri/.awless/keys/keypair-blog.pem’
OK keypair = keypair-blog
[info] Revert this template with `awless revert 01BHAE1C0CPMNYQ0KFPRQQ7XYA`

awless.io merges input from the command-line, user input and configurable defaults (for example, the image and instance type), and asks for confirmation before applying any modification to the infrastructure.

Now, we can easily create an instance using our brand new keypair:

> awless create instance keypair=@keypair-blog
Please specify (Ctrl+C to quit, Tab for completion):
instance.name? instance-blog
instance.subnet? <Tab>
@awless-private-subnet @awless-public-subnet @demo-env-subnet
instance.subnet? @demo-env-subnet
create instance count=1 image=ami-70edb016 keypair=keypair-blog name=instance-blog subnet=subnet-2486b17c type=t2.micro
Confirm? (y/n): y
[info] create tag ‘Name=instance-blog’ on ‘i-094ceffee40892f66’ done
[info] create instance ‘i-094ceffee40892f66’ done
OK instance = i-094ceffee40892f66
[info] Revert this template with `awless revert 01BHAGCXY80FPK5FR73GTWWSD2`

We can use another command, ssh, to directly login to that instance:

> awless ssh instance-blog
awless could not validate the authenticity of ‘34.252.193.183:22’ (unknown host)
ecdsa-sha2-nistp256 public key fingerprint is SHA256:Ay8ODMNQiEt1U6BoTsPyIajoaHZ047v7uNjTV1SSQmk.
Do you want to continue connecting and persist this key to ‘/Users/henri/.ssh/known_hosts’ (yes/no)? yes
[info] Login as ‘ec2-user’ on ‘34.252.193.183’, using keypair ‘/Users/henri/.awless/keys/keypair-blog.pem’ with ssh client ‘/usr/bin/ssh’
   __|  __|_  )
_| ( / Amazon Linux AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2016.09-release-notes/
18 package(s) needed for security, out of 52 available
Run “sudo yum update” to apply all updates.
Amazon Linux version 2017.03 is available.
[ec2-user@ip-10–0–20–245 ~]$

There is no need to pass keys as arguments, since they are stored locally, nor to specify the username as awless.io makes it own guess. The devil is in the detail, and awless.io strives to get the User Experience of CLIs right.

Beyond shortcuts, there are powerful new features at your fingertips. If you read carefully the output for previous commands, you noticed that each successful action has a revert id. Indeed, most of awless.io templates (one-liners or full scripts) can be reverted very easily.

Like git, awless.io maintains a log of all operations performed:

> awless log
[...]
ID: 01BHAE1C0CPMNYQ0KFPRQQ7XYA, Date: May 29 17:43:50, Author: user/hbinsztok, Region: eu-west-1
OK create keypair name=keypair-blog[keypair-blog]
ID: 01BHAGCXY80FPK5FR73GTWWSD2, Date: May 29 18:25:06, Author: user/hbinsztok, Region: eu-west-1
OK create instance count=1 image=ami-70edb016 keypair=keypair-blog name=instance-blog subnet=subnet-2486b17c type=t2.micro [i-094ceffee40892f66]

And like commits, operations can be reverted easily:

> awless revert 01BHAGCXY80FPK5FR73GTWWSD2
delete instance id=i-094ceffee40892f66
Confirm? (y/n): y
OK delete instance

Templates

Up to now, we used templates that contain a single command (one-liner). Templates are a generalization and a succession of awless.io commands:

# Title: Create an instance accessible with ssh with a new keypair
# Tags: infra, ssh

# Create a new security group for this instance

securitygroup = create securitygroup vpc={instance.vpc} description={securitygroup.description} name=ssh-from-internet
# Authorize access on port 22 to instances in this security group
update securitygroup id=$securitygroup inbound=authorize protocol=tcp cidr=0.0.0.0/0 portrange=22

# Create a new keypair
keypair = create keypair name={keypair.name}

# Create an instance in this security group accessible with the new keypair
create instance subnet={instance.subnet} image={instance.image} type={instance.type} keypair=$keypair name={instance.name} count=1 securitygroup=$securitygroup

The following features are added to chain commands together:

  • A command result can be stored in a value by using the syntax value = command, and the value can be later recalled as $value;
  • Template parameters are missing values in command arguments. They will be input at runtime and are written between braces, such as {instance.image} from the example above.

Templates are a very powerful feature. For example, they make it easy to complete tasks, such as:

To launch a template, you can use either repo: for the official repository awless-templates, a URL, or a local file:

> awless run repo:dynamic_autoscaling_watching_CPU

Try it for yourself!

Better yet, you will be able to use awless revert that we introduced earlier in this article to remove all the resources that are created by the template (instances, security group, target group, load balancer, etc.).

Note that the template implementation is still preliminary and might evolve with additional changes before awless.io reaches version 1.0. After that point, all language revisions will come with migration assistants.

Summary of commands

Remember that you can get help on all awless.io features by typing:

> awless help
> awless [verb] -h

The following tables list the main command verbs and template verbs:

Just the Beginning

awless.io 0.1.0 released today, following more than 20 alpha releases since the project publicly launched in January 2017.

The project is a work-in-progress. We expect awless.io to improve in many ways, including supporting more AWS services and features for each service. You’re welcome to file or upvote existing GitHub issues if your favorite service is (still) missing.

Our roadmap this year will focus on the following aspects:

  • 0.2.0 — Working with multiple regions. Using the local graph to suggest resources in their proper region, and switch them easily.
  • 0.3.0 — Improving the scripting language. We plan to define a formal semantics, and introduce type-checking of templates that will catch errors before performing any actual query to AWS (including as a dry run).
  • 0.4.0 — Leveraging the local graph to run (complex) queries on the global infrastructure.

We welcome you to contribute and help us spread the word! And to stay up-to-date on the latest news about awless.io, follow us on Twitter at @awlessCLI.