Speeding up AWS CodeBuild with Custom Build Environments

Sep 23, 2018 08:30 · 1595 words · 8 minute read DevOps Amazon AWS CodeBuild

If you’ve used CodeBuild for any substantial projects, you may have run into long build times. In some cases, this happens because your project’s build step takes a long time. There’s not much general advice I can give to improve that – you’re on your own there. In other cases, this happens because setting up the environment for building takes a long time. We can definitely improve that.

Automated build tools generally execute builds by provisioning a new system, deploying your source to it, and then running commands on it. The system it creates is the build environment used to build your project. Each provider creates build environments in a different way – some use VM’s, some use cloud instances, some reuse baremetal machines, some use Docker, and I’m sure some use a technology I’ve never heard of. Amazon’s CodeBuild uses Docker.

In order to simplify and speed up your builds, Amazon provides a number of pre-built Docker images used to create build environments. These images allow you to start with certain common languages and build tools available, and in many cases this is enough to get started. As your project grows, however, you may begin requiring additional packages on the system in order to build your project. There are a number of ways to accomplish this.

One solution for this is to add a pre-build step into the buildspec.yml used in your project. This file contains a list of commands that Amazon reads in order to figure out how to execute your build. By adding installation instructions for external software into your pre-build step, you can quickly test out your build and get it running. This technique seems to have widespread adoption, and it’s well supported.

Unfortunately, this approach has a downside: the pre-build steps get executed prior to every build. This can add significant delays. In some cases, it may cause your build to fail due to exceeding the timeout. You could solve this by increasing the build timeout, or you could solve this by baking your pre-build steps into a custom Docker image. Based on the title of this article, I’m sure you can guess which solution this post addresses.

Let’s do it.

Prerequisites

  • Docker
  • Terraform
  • AWS CLI with a default profile configured

Setting up a build

If you’ve read any of my other posts, you’ve probably figured out by now that I automate everything. This step is no exception. Let’s create a CodeBuild project using Terraform.

mkdir code-build-example
cd code-build-example

Since this post discusses a CodeBuild project in isolation rather than one in a complete environment, I’m going to skip a number of setup steps. If you plan to use this code in a real project, you’ll want to setup Terraform properly, which you can do by following the steps outlined here.

Create a file named terraform.tf with the following content:

provider "aws" {
  region     = "us-east-1"
}

resource "aws_iam_role" "build" {
  name = "build"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codebuild.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "build" {
  role = "${aws_iam_role.build.name}"

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "*"
      ],
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ]
    }
  ]
}
POLICY
}

resource "aws_codebuild_project" "build" {
  name          = "slow-build"
  description   = "A slow building CodeBuild project."
  build_timeout = "5"
  service_role  = "${aws_iam_role.build.arn}"

  artifacts {
    type = "NO_ARTIFACTS"
  }

  source {
    type            = "GITHUB"
    location        = "https://github.com/FindAPattern/code-build-custom-environment.git"
    git_clone_depth = 1
  }

  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image        = "aws/codebuild/nodejs:8.11.0"
    type         = "LINUX_CONTAINER"
  }
}

Let’s break this down. We’ve declared three Terraform resources:

  • An IAM role
  • An IAM role policy
  • A CodeBuild project

The CodeBuild project runs as an IAM role, which allows it to interact with a number of required AWS services, such as CloudWatch logs and S3. By default, it assumes the CodeBuild role, which already has a number of these required resources. We’ve attached an additional IAM policy to the role allowing it to write logs to CodeBuild (I can’t for the life of me figure out why this isn’t default behavior).

Once we have this new IAM role in place, we attach it to a new CodeBuild project. This project specifies a number of additional configuration options:

  • A name, description, and timeout
  • The storage location for built artifacts
  • The source repository’s location
  • The environment configuration

We’re mostly interested in the environment configuration, since this specifies the operating system, machine size, and build environment to use for our project. We’ve chosen a Linux-based Docker container, a small machine, and the NodeJS 8.11.0 image. Our builds will run in this environment.

Go ahead and create all of this by running:

terraform init
terraform apply

This will create the CodeBuild project. Let’s navigate to CodeBuild in AWS, run the newly created project, and see what happens.

Figure 1

Okay, so maybe this isn’t the best example. 46 seconds isn’t really that slow. Regardless, 37 of those seconds occurred during the pre-build phase. We can do better.

Let’s take a look at the buildspec.yml used for this build:

version: 0.2

phases:
  pre_build:
    commands:
      - echo Installing Ansible...
      - apt-get update
      - apt-get install -y software-properties-common
      - apt-add-repository ppa:ansible/ansible
      - apt-get update
      - apt-get install -y ansible
  build:
    commands:
      - ansible --version

It doesn’t do much. Before the build, it installs Ansible. The build phase simply prints Ansible’s version. Regardless, installing Ansible takes quite a bit of time. We can improve that by moving the installation into a custom build environment.

Creating a custom build environment

Amazon CodeBuild uses Docker containers as build environments. In order to create a custom build environemnt, we’ll need to:

  • Create a customer Docker image.
  • Store the image in a place accessible by CodeBuild.
  • Update the build environment in CodeBuild to pull the Docker image.

Piece of cake. Create a Dockerfile with the following content:

FROM ubuntu:xenial-20180123
 
USER root

RUN apt-get update \
    && apt-get upgrade --assume-yes \
    && apt-get install -y software-properties-common \
    && apt-add-repository ppa:ansible/ansible \
    && apt-get update \
    && apt-get install --no-install-recommends \
                       --assume-yes \
                       python2.7-minimal=2.7.12* \
                       python-pip=8.1.1-* \
                       unzip=6.0* \
                       wget=1.17.1* \
                       zip=3.0* \
                       ansible \
    && apt-get clean

RUN pip install --no-cache-dir \
    pip==9.0.1 \
    setuptools==38.4.0

RUN pip install --no-cache-dir awscli==1.14.30

This is admittedly a bit more complicated than necessary. I’ve taken it from another project and removed some of the custom steps. It basically just installs wget, zip, unzip, python2, pip, the AWS CLI, ansible, and then it cleans the cache. All we care about here is installing ansible, which it does.

Now we have to store this somewhere. To make this easy, let’s use Amazon AWS’s ECR container repository. As always, we’ll use Terraform.

Create a new file named buildenv.tf with the following content:

resource "aws_ecr_repository" "environments" {
  name = "build-image"
}

resource "aws_ecr_repository_policy" "environments" {
  repository = "${aws_ecr_repository.environments.name}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CodeBuildAccess",
      "Effect": "Allow",
      "Principal": {
        "Service": "codebuild.amazonaws.com"  
      },
      "Action": [
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:BatchCheckLayerAvailability"
      ]
    }
  ]
}
EOF
}

output "repository_url" {
  value = "${aws_ecr_repository.environments.repository_url}"
}

This Terraform file creates an ECR repository for the image, configures its permissions to allow access from CodeBuild, and then specifies an output variable containing the repository URL, which we’ll need for pushing images.

Do that now. Run the following command to login, build, and then push your Dockerfile to ECR:

aws ecr get-login --no-include-email | bash
docker build -t $(terraform output repository_url) .
docker push $(terraform output repository_url)

The first line logs into your new Docker repository.

The second line builds the image.

The third line pushes it to your repository.

Now let’s tie this all off. Open up your terraform.tf file and change:

environment {
  compute_type = "BUILD_GENERAL1_SMALL"
  image        = "aws/codebuild/nodejs:8.11.0"
  type         = "LINUX_CONTAINER"
}

…to…

environment {
  compute_type = "BUILD_GENERAL1_SMALL"
  image        = "${aws_ecr_repository.environments.repository_url}:latest"
  type         = "LINUX_CONTAINER"
}

Apply your changes by running:

terraform apply

That’s it! We’ve pushed our custom build environment and pointed a CodeBuild project at it.

Testing it out

Before we test this out, we’ll have to remove the pre-build phase from buildspec.yml. Unfortunately, I’ve used my own public repository in all the examples, so you won’t be able to do that without pointing the example at your own repository. If you’re up for that, you’ll want to change the buildspec.yml to the following:

version: 0.2

phases:
  build:
    commands:
      - ansible --version

Once you do that, go ahead and run your build twice (so that it downloads and caches the image). You should see a result that looks something like this:

Figure 2

We’re down from 46 seconds to 12 seconds! That’s great. For such a simple project, it’s a significant improvement. For a real project, you’ll likely see far better.

Well done.

Outro

If you haven’t noticed already, Amazon’s CodeBuild allows an incredibly amount of flexibility. You can use this technique to setup just about any complicated custom build environment. It even allows you to run the build containers in a specific VPC. As far as build environments go, there’s not much it can’t do.

This post may seem simple, and I suppose it is, but its value is substantial. It allows setting up highly customized continuous deployment in Amazon AWS. These custom build environments allow you to run tools like Terraform and Packer securely and efficiently, which we’ll get to soon.

That’s it for this week. As always, I’d love to hear your thoughts. Leave a comment, or send me an email by responding to my newsletter.

Happy coding!

You can get the final project’s sample code here.

Tweet Share

Subscribe to my newsletter to receive updates about new posts.

* indicates required