Setting up AWS Glue with Terraform

A quick Google search on how to get going with AWS Glue using Terraform came up dry for me. So, I went at it on my own and thought I’d share what I came up with (GitHubGist). Please note that I’m a Researcher by training — so this might not be the perfect solution for you. If you have recommendations on how to do this better, then please share in the comments below.

My example here will closely reflect the situation I was in. In particular, the S3 bucket I wanted to interact with was already defined and I didn’t want to give Glue full access to all of my buckets.

The first component is the role itself. Amazon recommends the particular name I use in this section so that the role can be passed from console users to the service. Check out the IAM Role Section of the Glue Manual in the References section if that isn’t acceptable. The other thing that was different from a boilerplate “Assume Role“ was “Principal” and “Service”.

resource "aws_iam_role" "glue" {
  name = "AWSGlueServiceRoleDefault"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

The next component was to attach the AWSGlueServiceRole managed policy to the role. Amazon pre-defines this so that the role has almost all of the permissions it needs in order to work out of the gate.

resource "aws_iam_role_policy_attachment" "glue_service" {
    role = "${aws_iam_role.glue.id}"
    policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"
}

If you don’t have a policy already defined for your S3 bucket, then you can define your policy and attach it to this glue role all in the same block, like this:

resource "aws_iam_role_policy" "my_s3_policy" {
  name = "my_s3_policy"
  role = "${aws_iam_role.glue.id}"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::my_bucket",
        "arn:aws:s3:::my_bucket/*"
      ]
    }
  ]
}
EOF
}

If, like me, you defined that policy but had already attached it to another role, then you can re-use it and attach it to the glue role as well like this:

resource "aws_iam_role_policy" "glue_service_s3" {
 name = "glue_service_s3"
    role = "${aws_iam_role.glue.id}"
    policy = "${aws_iam_role_policy.my_s3_policy.policy}"
}

The text you’d change here to match your configuration would be ‘my_s3_policy’ for the policy option/key.

Well, that’s it for now. Good luck!

References

• AWS Glue Manual, in particular the Create an IAM Role section
• Hashicorp’s Terraform Manual
• Liberal skimming of jamescarr’s other work using Terraform @Zapier. Seriously, he is one of my heros. =)