A Brief Introduction to Provisioning

Sep 9, 2018 13:30 · 2845 words · 14 minute read Infrastructure Provisioning Chef Ansible

So far we’ve discussed what’s in a production web application, and we’ve discussed how to create the physical hardware required to build one. This leaves us with two remaining problems:

  • How to install and configure software on each new server.
  • How to update and rollback application versions.

This post discusses how to install and configure software in our environment, a task often referred to as server provisioning.

Server Provisioning

Rather than jumping all the way to modern tools, let’s first consider the most basic and battle tested server provisioner: shell scripts. Before Chef or Ansible or Puppet, many operations teams provisioned servers using Bash or, if Windows is your thing, PowerShell scripts. This worked swimmingly for quite some time, and it still does.

For example, if we want to install Nginx on an Amazon EC2 instance running Ubuntu, we could use the following script (let’s call it: install-nginx.sh):

#!/bin/sh
ssh -t ubuntu@$1 sudo apt-get upgrade
ssh -t ubuntu@$1 sudo apt-get -y install nginx

If we had this script in the example given in previous article, we could have easily installed Nginx on our servers by running the following command:

./install-nginx.sh <ip of our instance>

Just kidding! We secured our machine in a private subnet, and we setup a firewall to prevent access via SSH. You can’t actually SSH into it without reconfiguring the security.

It’s important to understand that you could use shell scripts to provision everything on your servers. As far as I know, all main stream provisioning tools work by running shell commands over a transport layer like SSH or PowerShell (Chef might be an exception). Even if you’re using a provisioning tool, most likely you will have to do this at some point. For that reason alone, learning how to provision servers using basic shell scripting will pay many dividends once you start using a more polished provisioning tool like Chef or Ansible.

With that glowing introduction to shell provisioning, you might ask yourself why you should learn provisioning tools at all when you can already accomplish everything by running shell scripts over SSH. Plenty of environments already use shell scripts for provisioning, so what reason is there to replace them? It’s a valid question, or at least G. K. Chesterton thinks so:

In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.”

– G. K. Chesterton (https://www.chesterton.org/taking-a-fence-down/)

Perhaps this isn’t the most appropriate quote to use here, since you may not have built a “fence” yet. I’ve included it because I’ve personally found it helpful to remember when deciding whether or not to replace an older technology with a newer one. If shell scripts already solve these problems, why replace them with provisioning tools?

First, they generally use a declarative syntax. We discussed the benefits of this syntax in the previous post. Shell scripts install software by running an imperative sequence of commands; provisioning tools install software by specifying what software the server ought to have installed. This allows you to install and configure the same software across different operating systems, different package managers, and different versions using the same code.

Second, provisioning tools usually provide a way to organize infrastructure. While it’s possible to do this using shell scripts, the provisioning tool tends to force you into a design that’s clear and concise. It’s industry standard, so developers will have a much simpler task ahead of them when they try to figure out which servers in the QA environment run RabbitMQ.

Third, each major provisioning tool has a thriving community that builds reusable modules for installing most open source software. Rather than trying to remember where that Postgres config file lives, you can probably just stick the memory limit in the configuration for the module. This has saved me more time than it ought.

There’s probably quite a bit more to say here, but I’ll leave it at that for now. Despite the steep learning curve, learning provisioning tools is worth it. Compared to shell provisioning, it’s easier to write, easier to think about, and for those reasons it’s more maintainable.

Naming Things

The first few weeks of learning how to provision servers with Chef, a server provisioning tool, left a lasting impression on me. The getting started guide showed how to create a “recipe,” which contains instructions for installing or configuring a piece of software. I could get behind that analogy. Recipes must exist in a “cookbook.” That makes sense. You test cookbooks in the “kitchen.” I started looking around for hidden cameras.

This cooking analogy was so confusing that I decided to check out other tools, like Ansible. The first page of Ansible’s documentation introduced me to “playbooks.” Wait a minute. Playbooks contain a list of “plays.” Get out of here.

Okay. Why does this matter? It matters because you should know before learning a provisioning tool that they all seem to have a perplexing tendency to introduce vast amounts of jargon. You have to relearn definitions for quite a few words just to perform basic tasks. If you’re just getting started, I highly recommend writing down definitions as you go, because it’s quite a lot to learn.

In their defense, this isn’t all that uncommon in software. There’s a reason “naming things” is in the adage:

There are only two hard things in Computer Science: cache invalidation and naming things.

Phil Karlton

Every software developer creates alternative definitions for existing words. They also invent words, like “uninitialize” and “unregister.” It’s part of writing software. Sometimes we get it right, and sometimes we get it wrong. And sometimes we create fascinating, new, fictional universes in which we knife recipes from a cookbook into a node.

I’ll try to explain these tools using familiar terms. Once you understand the high level concepts in English, it’s a bit easier to map them to jargon.

Configuration Management

You’ve decided to install Nginx on your remote server using your fancy new provisioning tool. All goes well until you have to setup database backups. You’ve written your primary MySQL server’s config file, but you’re not quite sure how to populate the internal DNS address for your MySQL slave. That’s where configuration management enters the picture.

It’s helpful when setting up a server to think of every application as having two parts: an immutable portion (usually the code or compiled binaries), and a mutable portion (usually configuration files or environment variables). Most modules created by the community will by default install binaries and as much sensible configuration as it can, and they will expose attributes for you to override the configuration.

These attributes populating your configuration often contain values specific to your environment. Most provisioning tools provide mechanisms for you to insert environment-specific values into configuration files using templates, or directly into environment variables.

Using the configuration management provided by your provisioning tool, you can setup your MySQL master configuration file, which allows you to configure your slave.

Secret Management

This almost solves your problem, until you have to upload AWS credentials to allow your MySQL slave server access to S3. Good intuition prevented you from checking these credentials into your source repository, but that means the credentials live only on your machine and on NSA servers.

You need secret management.

As with all things in the world of automation, you have many choices for managing secrets. Google has a service called KMS, Amazon AWS has a service called Secret Manager, Chef has encrypted data bags, Hashicorp has a product called Vault, Ansible has a product called Vault, but I repeat myself. Aside from KMS, which really just encrypts strings, all of these tools perform the same function: securing access to encrypted secrets for use in configuration management.

I can’t tell you how many times I’ve run into secrets checked into repositories. Literally. If I told you the number of times I’ve run into this, I might get sued for violating some contract I signed a decade ago. It happens all the time, and it’s incredibly dangerous. I’ll leave you to decide what “it” means in that sentence.

Never store API keys or credentials in plaintext.

It’s worth repeating.

Never store API keys or credentials in plaintext.

Use a secret management solution to store this data, and then tie it into your provisioning tools.

That should be enough overview for now. Let’s build something.

A simple example: Chef

The example in the previous post used shell provisioning to install Nginx on a server created by Terraform. Let’s update that example to use Chef.

You’ll need to install the Chef Development Kit (ChefDK).

As mentioned numerous times already, we need a recipe to install Nginx. For instructional purposes, we’ll create it from scratch rather than use one from a community cookbook.

Speaking of cookbooks, you’ll need to create one. Cookbooks generally live in a cookbooks directory, so let’s create that first. From the root of your project, run the following command:

mkdir cookbooks

Now let’s create a cookbook in which to put our new recipe:

chef generate cookbook cookbooks/application

This command creates quite a few files inside of the cookbooks/application directory. The one we care about is: cookbooks/application/recipes/default.rb. This contains the default recipe for our cookbook. We’ll put commands to install Nginx into this file.

You can do that now. Add the following to cookbooks/application/recipes/default.rb:

apt_update

package 'nginx'

cookbook_file '/var/www/html/index.html' do
  source 'index.html'
  owner 'www-data'
  group 'www-data'
  mode '0755'
  action :create
end

The first two commands in this file do what you would expect:

  • apt_update updates your aptitude packages.
  • package 'nginx' installs the nginx package using your operating system’s default package manager (in our case, it uses aptitude).

The last command copies cookbooks/application/files/index.html into /var/www/html/index.html on your remote server. It also sets the file’s permissions to allow the Nginx server to access it.

This file doesn’t exist yet, so let’s create it. First, we’ll have to make the files directory. Run:

mkdir cookbooks/application/files

Then create a file at cookbooks/application/files/index.html with the following contents:

<html lang="en-us">
  <head>
    <title>Hello, World!</title>
  </head>
  <body>
    Chef has landed.
  </body>
</html>

Let’s update our packer.json with instructions to use Chef:

{
  "builders": [{
    "type": "amazon-ebs",
    "region": "us-east-1",
    "source_ami": "ami-04169656fea786776",
    "instance_type": "t2.small",
    "ssh_username": "ubuntu",
    "ami_name": "Ubuntu 16.04 Nginx - {{timestamp}}",
    "tags": {
      "Image": "application"
    }
  }],
  "provisioners": [{
    "type": "chef-solo",
    "cookbook_paths": ["cookbooks"],
    "run_list": ["recipe[application]"]
  }]
}

We’ve made two changes to the previous packer.json.

First, we’ve added an Image tag to the AMI. We previously copied and pasted the AMI ID out of Packer’s output and into our Terraform code. This isn’t a maintainable solution, since the AMI ID will change frequently, and we shouldn’t have to push a new change to our repository every time this happens. Instead, we’ll use a data resource in Terraform to read the AMI ID dynamically by querying for the most recent AMI with the Image=application tag.

Second, we’ve updated the shell provisioner to use chef-solo. You’ll notice that we’ve told it where to find our cookbooks directory, and we’ve told it which recipe to run. By default, the recipe[COOKBOOK] entry in a run_list executes the recipes/default.rb recipe. You can override this behavior by passing the recipe in explicity: recipe[COOKBOOK::RECIPE]. Since our recipe is stored in recipes/default.rb, we’ll use the default behavior.

Now let’s build our AMI. Run:

packer build packer.json

Our new AMI has an Image tag. Let’s change our hardcoded AMI’s in terraform.tf, our infrastructure definition, to lookup the AMI by tag.

Add the following to terraform.tf:

data "aws_ami" "web" {
  most_recent = true
  owners = ["self"]
  filter {                       
    name = "tag:Image"     
    values = ["application"]
  }                              
}

Now replace the AMI ID in your aws_instance.web1 and aws_instance.web2 resources with the output ID from your aws_ami.web resource:

resource "aws_instance" "web1" {
  ami                    = "${data.aws_ami.web.id}"
  availability_zone      = "us-east-1a"
  instance_type          = "t2.small"
  vpc_security_group_ids = ["${aws_security_group.application.id}"]
  subnet_id              = "${aws_subnet.private1.id}"
}

resource "aws_instance" "web2" {
  ami                    = "${data.aws_ami.web.id}"
  availability_zone      = "us-east-1b"
  instance_type          = "t2.small"
  vpc_security_group_ids = ["${aws_security_group.application.id}"]
  subnet_id              = "${aws_subnet.private2.id}"
}

That should do it. Run the following to create your Chef provisioned server, and then launch a browser to your load balancer’s domain name:

terraform plan -out terraform.plan
terraform apply "terraform.plan"
open "http://$(terraform output dns)"

You should see a webpage that reads: Chef has landed!

A simple example: Ansible

Note: This example relies on updates to terraform.tf made in the previous example. If you skipped it, make sure to go back and apply those changes.

Let’s build the exact same example using Ansible. You’ll need to install it to build this example.

Ansible organizes installation and configuration instructions into tasks. It organizes tasks inside of a playbook. Let’s create a directory structure for our playbook.

mkdir playbook
mkdir playbook/files

This is not the best practice for organizing Ansible playbooks. We’re using a simplified version due to our extremely trivial use case. If you’re interested in using Ansible as a provisioner, you should structure your playbook according to their recommendations, which you can read about here.

Let’s create our playbook inside of playbook/application.yml with the following content:

---
- hosts: all
  gather_facts: False
  become: yes
  pre_tasks:
  - name: Install Python 2.7
    raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)
- hosts: applications
  become: yes
  tasks:
  - name: Install Nginx
    apt:
      name: nginx
      state: present
      update_cache: yes
  - name: Update contents of index.html
    copy:
      src: index.html
      dest: /var/www/html/index.html
      owner: www-data
      group: www-data
      mode: 0755

This playbook file contains all of the information needed to provision our server. Let’s discuss its structure.

A playbook contains a list of “plays.” Each play contains a list of “tasks,” which it uses to install and configure software. In this case, our playbook contains two plays. The first play installs Python 2.7 on Ubuntu, which Ansible requires to run. The second play installs and configures Nginx.

At the root level of each play, we’ve configured two parameters: hosts and become. The hosts parameter tells Ansible on which machines it should run the playbook (“all” runs on all machines). become: yes causes Ansible to run all commands via sudo. Without it, you’ll get a lot of permissions errors.

The first task in the play for installing and configuring Nginx updates the aptitude cache, and then it guarantees the presence of the nginx package. It’s worth noting that if the nginx package is already installed, this command will do nothing.

The second task copies files/index.html into our remote server, and it assigns the correct permissions to it so that Nginx can serve the file.

That file doesn’t exist yet, so let’s create it. Put the following contents into: playbook/files/index.html:

<html lang="en-us">
  <head>
    <title>Hello, World!</title>
  </head>
  <body>
    Ansible has landed.
  </body>
</html>

That’s all we need to configure Ansible. Let’s tell Packer to use it. Update packer.json with the following contents:

{
  "builders": [{
    "type": "amazon-ebs",
    "region": "us-east-1",
    "source_ami": "ami-04169656fea786776",
    "instance_type": "t2.small",
    "ssh_username": "ubuntu",
    "ami_name": "Ubuntu 16.04 Nginx - {{timestamp}}",
    "tags": {
      "Image": "application"
    }
  }],
  "provisioners": [{
    "type": "ansible",
    "playbook_file": "./playbook/application.yml",
    "host_alias": "applications"
  }]
}

We’ve only changed the provisioner to use Ansible. This provisioner configuration requires a path to a playbook file, which we’ve set to ./playbook/application.yml. If you look at the play for installing Nginx, you’ll notice a hosts: applications line at the top. That’s the host alias we’re using to tell Ansible where to install our application. We need to tell Packer that we’re building an image for one of those hosts, so we set the host_alias attribute to applications.

That should do it. Run the following to create your Ansible provisioned server, and then launch a browser to your load balancer’s domain name:

packer build packer.json
terraform plan -out terraform.plan
terraform apply "terraform.plan"
open "http://$(terraform output dns)"

You should see a webpage that reads: Ansible has landed!

Outro

That should just about do it for an introduction to server provisioning. It’s been quite a long series of posts to get to this point, and there’s finally light at the end of the tunnel. Once again, I’ve underestimated the amount of complexity it required to provision servers, so we’ll have to leave continuous delivery for next week. I’ve also left out an example that shows how to manage secrets, so we’ll also have to do that in a future post.

Despite those omissions, I hope this post gave you a high level understanding of the concepts used in provisioning servers.

Happy coding!

The next post is live. You can read it here.

Tweet Share

Looking for graphic design help?

I partner with designers at Handshake Studios to provide graphic design feedback to freelancers and early-stage startups.

Sign up at DesignSavior.com!

Subscribe to my newsletter to receive updates about new posts.

* indicates required