Continuous Integration with Amazon CloudFormation via Jenkins

This post does not cover how to setup Jenkins. For information on how to install Jenkins see the official documentation.

Specifically, what we’re going to cover is:

  1. how to connect Jenkins to a git repo (on GitHub in this example)
  2. how to configure a git hook so that Jenkins runs jobs on git commit
  3. how to create a Jenkins job with Jenkins Job Builder
  4. how to use continuous integration on our Jenkins jobs
  5. a quick walkthrough on how to apply continuous integration on your CloudFormation stacks (in our example, the network stack)

This is one of my favorite topics and combines continuous integration with infrastructure as code concepts. There is a GitHub repository available with this post.

Connect Jenkins to a Git Repo

This assumes a working blank Jenkins server, but should work on most Jenkins installs.

You will need the GitHub plugin:

  1. Go to jenkins.example.com (where ever your Jenkins server exists).
  2. Click on “Manage Jenkins”. Or browse to jenkins.example.com/manage.
  3. Click on “Manage Plugins”. Or browse to jenkins.example.com/pluginManager/.
  4. Click on the “Available” tab. Or browse to jenkins.example.com/pluginManager/available.
  5. Filter by “GitHub”.
  6. Install the “GitHub Plugin”.

Next you’ll need to create a job and connect it the a repo. We’re going to use the GUI first, and then the Jenkins Job Builder. So the steps for the GUI are:

  1. Go to jenkins.example.com (where ever your Jenkins server exists).
  2. On the left hand side, click “New Item”.
  3. Give the project a reasonable name.
  4. Select “Freestyle project”.
  5. Click okay. The project configuration page will load.
  6. Add a reasonable description of what the job is doing. Even if it’s just you using the Jenkins server, it’s good to keep notes and comments in your work.
  7. Add a URL to the GitHub project. Example: https://github.com/666jfox777/example-jenkins-cloudformation
  8. If you need to include parameters for your command, check the “This build is parameterized” box, and add any needed parameters.
  9. Under source code management select Git.
  10. Add the repo URL. Example: https://github.com/666jfox777/example-jenkins-cloudformation.git
  11. For public repos you don’t need credentials. For private repos, add credentials as needed.
  12. Add any required build steps.
  13. Save / Apply.

Configure a Git Hook to push to Jenkins

This first step is easy - enable the build trigger for Build when a change is pushed to GitHub in your project.

The other change is more difficult, you need to enable GitHub to push to your Jenkins box. This requires some sort of external access to the Jenkins server - if you’re running on AWS EC2, this is simply a matter of granting a public ip, but for others you might need to set up DNAT or similar.

Assuming that you have external access to the Jenkins box configured you can go to your GitHub project and configure a webhook to push events to your Jenkins server. The steps for this are:

  1. Log into GitHub.
  2. Browse to your project page.
  3. Go to the projects settings page.
  4. Click on or view the “Webhooks & Services” page.
  5. Click “Add service”.
  6. Add the URL to your Jenkins server. Example: http://jenkins.example.com:8080/github-webhook/
  7. Click “Add service”.

If you make a commit now you should see it show up shortly in your Jenkins project’s “GitHub Hook Log”.

Using the Jenkins Job Builder with Jenkins

To configure the Jenkins Job Builder to work with Jenkins you need to:

  1. Install Jenkins Job Builder. See the official documentation or the OpenStack peice on Jenkins Job Builder.
  2. Create and configurate a settings file: ~/jenkins_jobs.ini
    \[job\_builder\]
    ignore\_cache=True
    include\_path=~/repo/jenkins\_jobs/
    recursive=True
        
    \[jenkins\]
    user=jenkins
    password=P@$$w0Rd
    url=https://jenkins.example.com
    

The bare basic template looks like:

\- job:
    name: job-name

We’ll need to build a template for our earlier job!

\- job:
    name: jenkins-job-builder
    project-type: freestyle
    defaults: global
    description: 'Updates Jenkins jobs from GitHub repo.'
    disabled: false
    display-name: 'Jenkins Job Builder'
    concurrent: true
    properties:
    - github:
        url: https://github.com/openstack-infra/jenkins-job-builder/
    scm:
    - git:
        skip-tag: false
        url: https://github.com/666jfox777/example-jenkins-cloudformation.git
    triggers:
    - github

You can test your template with: jenkins-jobs --conf jenkins_jobs.ini test job.yaml

You can run your template with: jenkins-jobs --conf jenkins_jobs.ini update test.yaml

If you’re having issues with writing a template for your job, the official documentation should be able to help you out.

Continuous Integration with CloudFormation

Integrating Jenkins with CloudFormation is relatively simple, you just need to add the following build steps:

\# Step 1: Locate your templates, and build them if necessary.
#cd ./source; ./cf-vpc-stack.py > ../build/cf-vpc-template.json

# Step 2: Copy the template to S3.
aws s3 sync ./build/ s3://${ACCOUNT}-cf-templates/\`date +%Y-%M-%d-%s\`;

# Step 3: Upload and update the stack. :)
aws cloudformation update-stack --stack-name cf --template-url https://s3.amazonaws.com/${ACCOUNT}-cf-templates/\`date +%Y-%M-%d-%s\`/cf-master-stacks.json --region ${REGION}

Technically the first and second steps are optional. We could include it all in one step by issuing aws cloudformation update-stack with the --template-body option instead of the --template-url option. However this compartmentalizes the process, allowing it to be completed in several smaller steps. Storing the processed template in AWS S3 also allows for you to cache versions of the build.

Note that CloudFormation stacks require the most number of IAM permissions if you’re using IAM roles. Be very cautious when granting these permissions - ensure your Jenkis server is appropriately protected!

Side note for those with multiple accounts - you’ll need to create a cross account IAM role on the second account and grant the first account access to the role. Once that’s done, Jenkins can assume the role during the job. Example:

\# Assume the right role
aws sts assume-role --role-arn "arn:aws:iam::${AWSID}:role/${ACCOUNT}-Jenkins" --role-session-name "Test" > assume-role-output.txt

# Export this stuff...
export AWS\_ACCESS\_KEY\_ID=\`cat assume-role-output.txt | jq -c '.Credentials.AccessKeyId' | tr -d '"' | tr -d ' '\`
export AWS\_SECRET\_ACCESS\_KEY=\`cat assume-role-output.txt | jq -c '.Credentials.SecretAccessKey' | tr -d '"' | tr -d ' '\`
export AWS\_SECURITY\_TOKEN=\`cat assume-role-output.txt | jq -c '.Credentials.SessionToken' | tr -d '"' | tr -d ' '\`