28 August 2014

In the previous post we discussed improving the speed of Travis CI builds by caching Maven dependencies in S3. In this post we’ll look at customising deployment to Heroku from Travis CI.

Travis CI supports deployment to Heroku with very little config. However, this approach wasn’t quite right for me for a couple of reasons:

• I wanted to deploy to Heroku during my build rather than at the end, so that I could run integration tests against my staging site
• I wanted the option of pushing the deployment to live in the case where the build succeeded

I also thought it might be possible to speed up the build a bit further by not re-building the application as part of the Heroku deploy, and not downloading the extra dependencies required by Travis CI’s Heroku deployment process.

Requirements

Here’s the detailed order of events I want to happen when I push a new version of the code:

• Build the application and run unit tests
• Deploy the application to Stage and run integration tests against it
• Mark the build as a success or failure based on the test results
• If the build is successful, then deploy the application to Live

The Travis CI build lifecycle

The Travis CI build lifecycle consists of the following steps:

• before_install
• install
• before_script
• script
• after_success or after_failure
• after_script
• deploy
• after_deploy

Each of these steps can be overridden by a custom script, except for the deploy step, which can only be configured to use one of Travis CI’s pre-defined deployment options.

To meet my requirements above, I’d need to deploy to Stage as part of the script step in order to pass or fail the build as a result, so can’t make use of the Heroku deployment feature provided by Travis CI. I could make use of this for deploying to Live, however having already built and deployed the application on Stage, it would be nice to push the built artifact to Live rather than having to build it again.

Using the Heroku API

Fortunately, Heroku support this exact scenario via its HTTP API. The process is as follows:

• Create a Heroku slug containing your application
• Deploy that slug to stage
• Deploy the same slug to Live

Creating a slug

A slug is just an archive containing your compiled application. Slugs are quite central to the Heroku build/deploy process but you don’t normally see them if using Heroku’s standard git-based deployment, since they are created and released automatically for you. Here’s what Heroku does for you when you push your code via git:

• Determines which build pack to use based on your project (e.g. the Java buildpack if you have a pom.xml)
• Creates a slug using the appropriate buildpack
• Deploys this slug to your application

We’re kicking off our deployment from our CI server, so have already just compiled the application. This makes it easy to create a slug for our code. We just need a tar archive with all the contents under the /app path (which will be the base path of our application on the Heroku server):

echo "Preparing slug contents"
mkdir app
cp -r ./exwhy-web/target/classes ./app/classes
cp -r ./exwhy-web/target/lib ./app/lib
 
echo "Creating slug archive"
tar -czf slug.tgz ./app

The next stage is to create a slug containing this archive. First we call the Heroku API to create a new slug. This returns an S3 URL to which we upload our archive. Lets look at the slug creation first:

echo "Creating slug object"
_heroku_deploy_apiKey=`echo ":${HEROKU_API_KEY}" | base64`
_heroku_deploy_createSlugResponse=$(curl -X POST \
-H "Content-Type: application/json" \
-H "Accept: application/vnd.heroku+json; version=3" \
-H "Authorization: ${_heroku_deploy_apiKey}" \
-d '{"process_types":{"web": "java $JAVA_OPTS -cp ./classes:./lib/* io.hgc.myapp.web.Application"}}' \
-n https://api.heroku.com/apps/${HEROKU_STAGE}/slugs)

We transform our API key into the right format and then make a POST to https://api.heroku.com/apps/appName/slugs to create a new slug. One thing to note here is request data (the -d parameter), specifying the web process. When using git-based deployment you would normally specify your processes in a Procfile. This file isn’t built into the slug but becomes part of its metadata, so when creating a slug from scratch we have to specify that metadata ourselves rather than using a Procfile.

The response from the above request will look something like the following:

{
  "blob": {
    "method": "put",
    "url": "https:\/\/s3-external-1.amazonaws.com\/herokuslugs\/heroku.com\/v1\/some-guid?credentials"
  },
  "buildpack_provided_description": null,
  "commit": null,
  "created_at": "2014-08-25T08:32:15Z",
  "id": "9f6f132a-5a87-423b-b677-84c7a35a42c5",
  "process_types": {
    "web": "java $JAVA_OPTS -cp .\/classes:.\/lib\/* io.hgc.myapp.web.Application"
  },
  "size": null,
  "updated_at": "2014-08-25T08:32:15Z",
  "stack": {
    "id": "7e04461d-ec81-4bdd-8b37-b69b320a9f83",
    "name": "cedar"
  }
}

The important things for us are the ID of the slug and the blob URL. First, we need to extract these from the response (grep is really not an appropriate tool for parsing JSON, but the response is very simple so we get away with it):

function _heroku_deploy_parseField {
    echo -ne $2 | grep -o "\"$1\"\s*:\s*\"[^\"]*\"" | head -1 | cut -d '"' -f 4
}
_heroku_deploy_blobUrl=$(_heroku_deploy_parseField "url" "'${_heroku_deploy_createSlugResponse}'")
_heroku_deploy_blobHttpMethod=$(_heroku_deploy_parseField "method" "'${_heroku_deploy_createSlugResponse}'")
_heroku_deploy_slugId=$(_heroku_deploy_parseField "id" "'${_heroku_deploy_createSlugResponse}'")

Next we use the URL to upload our slug contents. The only wrinkle here is to make sure we put the HTTP method into uppercase, else the AWS signature check fails:

echo "Uploading slug archive"
curl -X ${_heroku_deploy_blobHttpMethod^^} -H "Content-Type:" --data-binary @slug.tgz ${_heroku_deploy_blobUrl}

Deploying the slug to stage

Finally, we create a new release of our application using the ID of our slug:

function deployToHeroku { #Args: application name
    curl -X POST \
    -H "Content-Type: application/json" \
    -H "Accept: application/vnd.heroku+json; version=3" \
    -H "Authorization: ${_heroku_deploy_apiKey}" \
    -d "{\"slug\":\"${_heroku_deploy_slugId}\"}" \
    -n https://api.heroku.com/apps/$1/releases
}

echo "Deploying slug to stage"
deployToHeroku ${HEROKU_STAGE}

Deploying the slug to Live

All we need to do to push the same slug to Live is to call deployToHeroku again with a different app name.

Putting this all together, we use the script in our .travis.yml file as follows:

script: mvn clean install && source ./build/heroku_deploy.sh && mvn test -Dtest.server=http://$HEROKU_STAGE.herokuapp.com
after_success: deployToHeroku $HEROKU_LIVE
env:
  global:
  - HEROKU_STAGE: myapp-test
  - HEROKU_LIVE: myapp
  ...

A couple of things to note here: Firstly, the test.server parameter is a custom parameter used by my test code, but you would probably want to do something similar if running tests against a staging server. Secondly, the various commands in the script step are all combined into one line with &&, rather than placed on separate lines. This is to make our build fail fast as soon as any one of them returns an error code, as described in the Travis CI documentation on customising the build.

Running the correct Java version

All of the above works perfectly, except for one thing: My application is built using Java 8, and the Heroku servers run Java 6 by default. When using Heroku’s git-based deployment, we would address this by adding a system.properties file in our project root containing java.runtime.version=1.8. However, like the ProcFile, this isn’t something that’s used at runtime but at build time in creating our slug.

I spent a while digging through the code of the Heroku Java buildpack and JVM common buildpack functions to see how they worked. They actually download OpenJDK and include it in the slug archive. They’re also pretty complicated due to being quite general-purpose.

I could have imitated the relevant parts of these scripts making some simplifying assumptions, or downloaded and used the JVM common buildpack functions directly in my script. However, it occurred to me that there’s a much simpler way to get hold of a JRE of the right version for our application: We can just re-use the JDK from our build environment! All we need to do is add a couple of lines to our slug creation script:

cp -r ${JAVA_HOME}/jre ./app/.jre
mkdir "app/.profile.d" && echo 'export PATH="/app/.jre/bin:$PATH"' >> app/.profile.d/java.sh

The second line adds a script under the .profile.d folder, which means it will run on startup. All the script does is add the JRE’s bin folder to the front of the $PATH so that it gets used in preference to the JDK 1.6 installed on the system.

Conclusion

With the changes in this post and the previous one, my build is now working exactly as I wanted, and much faster than before, down from its original time of 5-8 minutes to more like 1-2 minutes (usually closer to one minute). The most variable part of the build now is waiting for the new version of the application to start up on the staging server in order to run tests against it, which can take anything from ~20s to about a minute. However, the build doesn’t waste any time waiting for Heroku to build the application from scratch.

You can find the complete and final script on GitHub in the context of a demo Java application.