Building images with Packer

We're going to cover:

  • Downloading and installing Packer
  • Creating a basic template.
  • Packer commandline options - Inspect, Validate, Build
  • Combining Packer with Jenkins
  • Creating images for multiple environments

Installing Packer

In order to install packer you'll first need to download it. For CentOS and other Linux distributions:

wget https://dl.bintray.com/mitchellh/packer/packer_0.7.5_linux_amd64.zip

Then unpack it:

unzip packer_*.zip

Next we need to move the executables somewhere useful:

mv packer_*/* /usr/bin/

And now you should be able to verify the installation with:

packer

And the help menu should print out.

Packer Templates

A packer template is just a JSON formatted file with the correct fields provided. Valid top level fields are:

  • description:
    The description field is a string providing a description of what the template does. This output is used only in the command packer inspect. A good description is a good way to add value to a template - it helps when you or a teammate look at the template next.

  • variables:
    The variables field is an array of one or more key/value strings that defines user variables contained in the template. If it is not specified, then no variables are defined. Variables should be favored over hard coding, and allow a template to be reused.

  • builders:
    The builders field is an array of one or more objects that defines the builders that will be used to create machine images for this template, and configures each of those builders. For example, there are separate builders for EC2, VMware, VirtualBox, etc. Packer comes with many builders by default, and can also be extended to add new builders. This is the only required field.

  • provisioners:
    The provisioners field is an array of one or more objects that defines the provisioners that will be used to install and configure software for the machines created by each of the builders. If it is not specified, then no provisioners will be run. Provisioners include scripts, file uploads, and SaltStack.

  • post-processors:
    The post-processors field is an array of one or more objects that defines the various post-processing steps to take with the built images. If not specified, then no post-processing will be done. A common usage of a post-processor is to build a vagrant box.

  • min_packer_version:
    The min_packer_version field is a string that has a minimum Packer version that is required to parse the template. This can be used to ensure that proper versions of Packer are used with the template.

A bare minimal template looks like this:

{
    "builders": [...]
}

Where the ... could be expanded to a Virtual Box builder and/or an Amazon EBS AMI builder:

{
    "builders": [
        {
            "type": "virtualbox-iso",
            "guest_os_type": "RedHat_64",
            "iso_url": "http://centos.mirrors.atwab.net/6.6/isos/x86_64/CentOS-6.6-x86_64-minimal.iso",
            "iso_checksum": "5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d",
            "iso_checksum_type": "sha256",
            "ssh_username": "vagrant",
            "ssh_password": "vagrant",
            "ssh_wait_timeout": "30s",
            "shutdown_command": "echo 'packer' | sudo -S shutdown -P now"
        },
        {
            "type": "amazon-ebs",
            "access_key": "YOUR KEY HERE",
            "secret_key": "YOUR SECRET KEY HERE",
            "region": "us-west-2",
            "source_ami": "ami-de0d9eb7",
            "instance_type": "t1.micro",
            "ssh_username": "vagrant",
            "ami_name": "packer-base {{timestamp}}"
        }
    ]
}

To give a better example, let's build one piece by piece to accomplish a specific scenario. Let's define our scenario as:

  • We want a CentOS image that we can use for local development in VirtualBox (or with Vagrant).
  • We want a CentOS-like image that we can use for production in Amazon. (For the purpose of this document, I'll be using a current Amazon Linux AMI)
  • We want our image to be fully patched and up-to-date.
  • We want our image to have certain packages pre-installed.
  • We want our software pre-installed.
  • We want to use SaltStack to install and configure our base image.

I think that those are some good goals to begin with.

Let's start with the first goal and get a CentOS box ready for Vagrant (with VirtualBox as the hypervisor):

{
    "builders": [
        {
            "type": "virtualbox-iso",
            "guest_os_type": "RedHat_64",
            "iso_url": "http://centos.mirrors.atwab.net/6.6/isos/x86_64/CentOS-6.6-x86_64-minimal.iso",
            "iso_checksum": "5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d",
            "iso_checksum_type": "sha256",
            "ssh_username": "vagrant",
            "ssh_password": "vagrant",
            "ssh_wait_timeout": "30s",
            "shutdown_command": "echo 'packer' | sudo -S shutdown -P now"
        }
    ]
}

If you ran packer build against this template you would not get much. The build would fail eventually since we didn't pass anything to the installer. But it runs and does something! Awesome!

Now the mirror I'm using above probably wouldn't make sense for you - so lets make this more dynamic by adding in variables!

{
    "description" : "Example CentOS 6.6 packer template",
    "variables": {
        "mirror": "http://centos.mirrors.atwab.net",
        "os_ver": "6.6"
    },
    "builders": [
        {
            "type": "virtualbox-iso",
            "guest_os_type": "RedHat_64",
            "iso_url": "{{user `mirror`}}/{{user `os_ver`}}/isos/x86_64/CentOS-{{user `os_ver`}}-x86_64-minimal.iso",
            "iso_checksum": "5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d",
            "iso_checksum_type": "sha256",
            "ssh_username": "vagrant",
            "ssh_password": "vagrant",
            "ssh_wait_timeout": "30s",
            "shutdown_command": "echo 'packer' | sudo -S shutdown -P now"
        }
    ]
}

Now lets see if we can get a basic image working. We need:

  • basic build parameters for virtual box
  • kickstart script to configure CentOS
  • provisioning scripts / saltstack

We'll do these in order so first let's add some basic build parameters to our CentOS box.

{
    "description" : "Example CentOS 6.6 packer template",
    "variables": {
        "mirror": "http://centos.mirrors.atwab.net",
        "os_ver": "6.6"
    },
    "builders": [
        {
            "type": "virtualbox-iso",
            "guest_os_type": "RedHat_64",
            "iso_url": "{{user `mirror`}}/{{user `os_ver`}}/isos/x86_64/CentOS-{{user `os_ver`}}-x86_64-minimal.iso",
            "iso_checksum": "5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d",
            "iso_checksum_type": "sha256",
            "guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
            "virtualbox_version_file": ".vbox_version",
            "http_directory": "http",
            "boot_command": [
                " text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg"
            ],
            "boot_wait": "10s",
            "disk_size": 10240,
            "vboxmanage": [
                [ "modifyvm", "{{.Name}}", "--memory", "512" ],
                [ "modifyvm", "{{.Name}}", "--cpus", "1" ]
            ],
            "ssh_username": "vagrant",
            "ssh_password": "vagrant",
            "ssh_wait_timeout": "10000s",
            "ssh_port": 22,
            "shutdown_command": "echo 'packer' | sudo -S shutdown -P now"
        }
    ]
}

For the most part what I've added is fairly obvious - disk, cpu, memory, ssh details... But some of it is not. To make the virtual machine work properly with VirtualBox, we pass in the VirtualBox Guest Additions package. To get the image to boot and become a usable virtual machine we create a webserver that packer will use (http_directory) and supply a boot command.

In the boot command you'll notice .HTTPIP and .HTTPPort. These will be generated by packer due to the usage of http_directory.

The kickstarter script used in this example is:

./http/ks.cfg:

install
cdrom
lang en_US.UTF-8
keyboard us
network --onboot yes --device eth0 --bootproto dhcp --noipv6
rootpw --plaintext vagrant
firewall --enabled --service=ssh
authconfig --enableshadow --passalgo=sha512
selinux --disabled
timezone America/Vancouver
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"

text skipx zerombr

clearpart –all –initlabel autopart

auth –useshadow –enablemd5 firstboot –disabled reboot

%packages –ignoremissing @core bzip2 kernel-devel kernel-headers -ipw2100-firmware -ipw2200-firmware -ivtv-firmware %end

%post /usr/bin/yum -y install sudo /usr/sbin/groupadd -g 501 vagrant /usr/sbin/useradd vagrant -u 501 -g vagrant -G wheel echo “vagrant”|passwd –stdin vagrant echo “vagrant ALL=(ALL) NOPASSWD: ALL” » /etc/sudoers.d/vagrant chmod 0440 /etc/sudoers.d/vagrant %end

You can read more on kickstart files and the available options here. It’s important to note that we could have done the kickstart commands inside the boot_command option.

Since we should now have a working builder, lets add a post processor.

{
“description” : “Example CentOS + SaltStack packer template”,
“variables”: {
“mirror”: “http://centos.mirrors.atwab.net”,
“os_ver”: “6.6”
},
“builders”: [
{
“type”: “virtualbox-iso”,
“guest_os_type”: “RedHat_64”,
“iso_url”: “{{user mirror}}/{{user os_ver}}/isos/x86_64/CentOS-{{user os_ver}}-x86_64-minimal.iso”,
“iso_checksum”: “5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d”,
“iso_checksum_type”: “sha256”,
“guest_additions_path”: “VBoxGuestAdditions_{{.Version}}.iso”,
“virtualbox_version_file”: “.vbox_version”,
“http_directory”: “http”,
“boot_command”: [
“ text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg”
],
“boot_wait”: “10s”,
“disk_size”: 10240,
“vboxmanage”: [
[ “modifyvm”, “{{.Name}}”, “–memory”, “512” ],
[ “modifyvm”, “{{.Name}}”, “–cpus”, “1” ]
],
“ssh_username”: “vagrant”,
“ssh_password”: “vagrant”,
“ssh_wait_timeout”: “10000s”,
“ssh_port”: 22,
“shutdown_command”: “echo ‘packer’ | sudo -S shutdown -P now”
}
],
“post-processors”: [
{
“type”: “vagrant”,
“override”: {
“virtualbox”: {
“output”: “centos-6-6-x64-virtualbox.box”
}
}
}
]
}

Before we look at adding SaltStack to this build process, let’s look at how to run basic scripts.

First we’ll need to add a provisioners array section, with a script provisioner defined. We’ll work with the remote shell provisioner - for more details, see the documentation.

{
“description” : “Example CentOS + SaltStack packer template”,
“variables”: {
“mirror”: “http://centos.mirrors.atwab.net”,
“os_ver”: “6.6”
},
“builders”: [
{
“type”: “virtualbox-iso”,
“guest_os_type”: “RedHat_64”,
“iso_url”: “{{user mirror}}/{{user os_ver}}/isos/x86_64/CentOS-{{user os_ver}}-x86_64-minimal.iso”,
“iso_checksum”: “5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d”,
“iso_checksum_type”: “sha256”,
“guest_additions_path”: “VBoxGuestAdditions_{{.Version}}.iso”,
“virtualbox_version_file”: “.vbox_version”,
“http_directory”: “http”,
“boot_command”: [
“ text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg”
],
“boot_wait”: “10s”,
“disk_size”: 10240,
“vboxmanage”: [
[ “modifyvm”, “{{.Name}}”, “–memory”, “512” ],
[ “modifyvm”, “{{.Name}}”, “–cpus”, “1” ]
],
“ssh_username”: “vagrant”,
“ssh_password”: “vagrant”,
“ssh_wait_timeout”: “10000s”,
“ssh_port”: 22,
“shutdown_command”: “echo ‘packer’ | sudo -S shutdown -P now”
}
],
“post-processors”: [
{
“type”: “vagrant”,
“override”: {
“virtualbox”: {
“output”: “centos-6-6-x64-virtualbox.box”
}
}
}
],
“provisioners”: [
{
“type”: “shell”,
“scripts”: [
“scripts/base.sh”,
“scripts/vagrant.sh”,
“scripts/virtualbox.sh”,
“scripts/cleanup.sh”
],
“only”: [“virtualbox-iso”]
},
]
}

For simple organization I’ve placed the scripts under ./scripts/*. You can view the scripts here.

At this point it should be safe to run packer validate packer.json and packer build packer.json.

It’s fairly easy to add a salt provisioner:

{
“description” : “Example CentOS + SaltStack packer template”,
“variables”: {
“mirror”: “http://centos.mirrors.atwab.net”,
“os_ver”: “6.6”
},
“builders”: [
{
“type”: “virtualbox-iso”,
“guest_os_type”: “RedHat_64”,
“iso_url”: “{{user mirror}}/{{user os_ver}}/isos/x86_64/CentOS-{{user os_ver}}-x86_64-minimal.iso”,
“iso_checksum”: “5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d”,
“iso_checksum_type”: “sha256”,
“guest_additions_path”: “VBoxGuestAdditions_{{.Version}}.iso”,
“virtualbox_version_file”: “.vbox_version”,
“http_directory”: “http”,
“boot_command”: [
“ text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg”
],
“boot_wait”: “10s”,
“disk_size”: 10240,
“vboxmanage”: [
[ “modifyvm”, “{{.Name}}”, “–memory”, “512” ],
[ “modifyvm”, “{{.Name}}”, “–cpus”, “1” ]
],
“ssh_username”: “vagrant”,
“ssh_password”: “vagrant”,
“ssh_wait_timeout”: “10000s”,
“ssh_port”: 22,
“shutdown_command”: “echo ‘packer’ | sudo -S shutdown -P now”
}
],
“post-processors”: [
{
“type”: “vagrant”,
“override”: {
“virtualbox”: {
“output”: “centos-6-6-x64-virtualbox.box”
}
}
}
],
“provisioners”: [
{
“type”: “shell”,
“scripts”: [
“scripts/base.sh”,
“scripts/vagrant.sh”,
“scripts/virtualbox.sh”,
“scripts/cleanup.sh”
],
“only”: [“virtualbox-iso”]
},
{
“type”: “salt-masterless”,
“local_state_tree”: “./salt”,
“local_pillar_roots”: “./pillar”
}
]
}

You can place any salt states you want to run in ./salt/ and ./pillar/.

From our goals earlier our only outstanding objective is building our Amazon AMIs for use on EC2.

{
“description” : “Example CentOS + SaltStack packer template”,
“variables”: {
“mirror”: “http://centos.mirrors.atwab.net”,
“os_ver”: “6.6”,
“access_key”: “”,
“secret_key”: “”,
“region”: “us-west-2”,
“vpc”: “”,
“subnet_id”: “”,
“source_ami”: “ami-e7527ed7”,
“instance_type”: “m3.medium”,
“ssh_username”: “root”,
“account_id”: “”,
“security_group_id”: “”,
“iam_instance_profile”: “”,
“s3_bucket”: “”,
“x509_cert_path”: “~/.aws/certificate.pem”,
“x509_key_path”: “~/.aws/private-key.pem”
},
“builders”: [
{
“type”: “virtualbox-iso”,
“guest_os_type”: “RedHat_64”,
“iso_url”: “{{user mirror}}/{{user os_ver}}/isos/x86_64/CentOS-{{user os_ver}}-x86_64-minimal.iso”,
“iso_checksum”: “5458f357e8a55e3a866dd856896c7e0ac88e7f9220a3dd74c58a3b0acede8e4d”,
“iso_checksum_type”: “sha256”,
“guest_additions_path”: “VBoxGuestAdditions_{{.Version}}.iso”,
“virtualbox_version_file”: “.vbox_version”,
“http_directory”: “http”,
“boot_command”: [
“ text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg”
],
“boot_wait”: “10s”,
“disk_size”: 10240,
“vboxmanage”: [
[ “modifyvm”, “{{.Name}}”, “–memory”, “512” ],
[ “modifyvm”, “{{.Name}}”, “–cpus”, “1” ]
],
“ssh_username”: “vagrant”,
“ssh_password”: “vagrant”,
“ssh_wait_timeout”: “10000s”,
“ssh_port”: 22,
“shutdown_command”: “echo ‘packer’ | sudo -S shutdown -P now”
},
{
“type”: “amazon-instance”,
“access_key”: “{{user access_key}}”,
“secret_key”: “{{user secret_key}}”,
“region”: “{{user region}}”,
“vpc_id”: “{{user vpc}}”,
“subnet_id”: “{{user subnet_id}}”,
“source_ami”: “{{user source_ami}}”,
“instance_type”: “{{user instance_type}}”,
“ssh_username”: “{{user ssh_username}}”,

        "ami_virtualization_type": "hvm",
        "bundle_vol_command": "sudo -E -n /opt/aws/bin/ec2-bundle-vol -k {{.KeyPath}} -u {{.AccountId}} -c {{.CertPath}} -r {{.Architecture}} -e {{.PrivatePath}} -d {{.Destination}} -p {{.Prefix}} --no-filter --batch",
        "bundle_upload_command": "sudo -E -n /opt/aws/bin/ec2-upload-bundle -b {{.BucketName}} -m {{.ManifestPath}} -a {{.AccessKey}} -s {{.SecretKey}} -d {{.BundleDirectory}} --batch --retry",
        "bundle_destination": "/media/ephemeral0",

        "account_id": "{{user `account_id`}}",
        "security_group_id": "{{user `security_group_id`}}",
        "iam_instance_profile": "{{user `iam_instance_profile`}}",

        "s3_bucket": "{{user `s3_bucket`}}",
        "x509_cert_path": "{{user `x509_cert_path`}}",
        "x509_key_path": "{{user `x509_key_path`}}",
        "x509_upload_path": "/tmp",
        "enhanced_networking": "true",

        "ami_name": "packer aws-is {{timestamp}}"
    },
    {
        "type": "amazon-ebs",
        "access_key": "{{user `access_key`}}",
        "secret_key": "{{user `secret_key`}}",
        "region": "{{user `region`}}",
        "vpc_id": "{{user `vpc`}}",
        "subnet_id": "{{user `subnet_id`}}",
        "source_ami": "{{user `source_ami`}}",
        "instance_type": "{{user `instance_type`}}",
        "ssh_username": "{{user `ssh_username`}}",

        "ami_virtualization_type": "hvm",
        "bundle_vol_command": "sudo -E -n /opt/aws/bin/ec2-bundle-vol -k {{.KeyPath}} -u {{.AccountId}} -c {{.CertPath}} -r {{.Architecture}} -e {{.PrivatePath}} -d {{.Destination}} -p {{.Prefix}} --no-filter --batch",
        "bundle_upload_command": "sudo -E -n /opt/aws/bin/ec2-upload-bundle -b {{.BucketName}} -m {{.ManifestPath}} -a {{.AccessKey}} -s {{.SecretKey}} -d {{.BundleDirectory}} --batch --retry",
        "bundle_destination": "/media/ephemeral0",

        "account_id": "{{user `account_id`}}",
        "security_group_id": "{{user `security_group_id`}}",
        "iam_instance_profile": "{{user `iam_instance_profile`}}",

        "s3_bucket": "{{user `s3_bucket`}}",
        "x509_cert_path": "{{user `x509_cert_path`}}",
        "x509_key_path": "{{user `x509_key_path`}}",
        "x509_upload_path": "/tmp",
        "enhanced_networking": "true",

        "ami_name": "packer aws-ebs {{timestamp}}"
    }
],
"post-processors": [
    {
      "type": "vagrant",
      "override": {
            "virtualbox": {
                "output": "centos-6-6-x64-virtualbox.box"
            }
        }
    }
],
"provisioners": [
    {
        "type": "shell",
        "scripts": [
            "scripts/base.sh",
            "scripts/vagrant.sh",
            "scripts/virtualbox.sh",
            "scripts/cleanup.sh"
        ],
        "only": ["virtualbox-iso"]
    },
    {
        "type": "shell",
        "inline": [
            "sudo echo \"helloworld\""
        ]
    },
    {
        "type": "salt-masterless",
        "local_state_tree": "./salt",
        "local_pillar_roots": "./pillar"
    }
]

}

Of main interest here are the amazon-instance and amazon-ebs builders. Notice that the sections are identical other than the specification of which builder to use.

Inspect, Validate, Build

Now that we have a working template we’re going to look at three commands that we can run against it. Those commands are:

  • packer inspect:
    The packer inspect Packer command takes a template and outputs the various components a template defines. This can help you quickly learn about a template without having to dive into the JSON itself. The command will tell you things like what variables a template accepts, the builders it defines, the provisioners it defines and the order they’ll run, etc.

  • packer validate:
    The packer validate Packer command is used to validate the syntax and configuration of a template. The command will return a zero exit status on success, and a non-zero exit status on failure. Additionally, if a template doesn’t validate, any error messages will be outputted.

  • packer build:
    The packer build Packer command takes a template and runs all the builds within it in order to generate a set of artifacts. The various builds specified within a template are executed in parallel, unless otherwise specified. And the artifacts that are created will be outputted at the end of the build.

In practice you’ll probably mostly use the packer build command most of the time, although the other two commands do provide some value.

Output from our template:

$ packer inspect packer.json
Description:

Example CentOS + SaltStack packer template

Optional variables and their defaults:

access_key = account_id = iam_instance_profile = instance_type = m3.medium mirror = http://centos.mirrors.atwab.net os_ver = 6.6 region = us-west-2 s3_bucket = secret_key = security_group_id = source_ami = ami-e7527ed7 ssh_username = root subnet_id = vpc = x509_cert_path = ~/.aws/certificate.pem x509_key_path = ~/.aws/private-key.pem

Builders:

amazon-ebs
amazon-instance virtualbox-iso

Provisioners:

shell shell salt-masterless

$ packer validate -only=virtualbox-iso packer.json Template validated successfully.

Assuming that the other variables are okay, just need to specify the blanks.

$ packer validate -only=amazon-instance
-var ‘access_key=VAR’
packer.json

$ packer build -only=virtualbox-iso packer.json virtualbox-iso output will be in this color.

==> virtualbox-iso: Downloading or copying Guest additions virtualbox-iso: Downloading or copying: file:///Applications/VirtualBox.app/Contents/MacOS/VBoxGuestAdditions.iso ==> virtualbox-iso: Downloading or copying ISO … virtualbox-iso (vagrant): Renaming the OVF to box.ovf… virtualbox-iso (vagrant): Compressing: Vagrantfile virtualbox-iso (vagrant): Compressing: box.ovf virtualbox-iso (vagrant): Compressing: metadata.json virtualbox-iso (vagrant): Compressing: packer-virtualbox-iso-1435602075-disk1.vmdk Build ‘virtualbox-iso’ finished.

Packer + Jenkins

While you could use packer on each developers machine, etc, it’s probably better and more convenient to have it installed to a central server - such as jenkins - and from there build and upload your images.

Creating a Jenkins job to run packer is fairly simple - we just need to ensure that the packer binaries are installed and that Jenkins has access to our template (and supporting configuration files / scripts).

But we have options now - how do we get the images from Jenkins? My preferred method is to upload them from Jenkins to S3, and have developers download the most recent image from S3 when they want it. Also we would have the images become AMIs from Jenkins as well.