computers, programming, startups and life.

Running Travis CI tests on ARM architecture

In my previous post, I have described how I got Luvit and Rackspace Monitoring Agent to build and run on Raspberry Pi.

If you can recall, one of the reasons why Luvit failed to build on ARM was that things got out of sync and there was no automatic process in place which would catch those issues by automatically trying to build and run the tests on Raspberry Pi / ARM.

Luckily, in this case the problem was a limitation of our continuous integration platform and not a lack of tests (Luvit has a pretty substantial test suite).

Luvit use Travis CI to automatically run the tests. Travis CI is a great service (more on that bellow), but it only supports running tests on x64 architecture¹.

To prevent similar issues from happening again in the future, I decided to get our tests to run on ARM on Travis CI. This post contains instructions on how you can do the same for your project.

Travis CI

If you aren’t familiar with it yet, Travis CI is an awesome free continuous integration service for your open-source projects². It allows you to automatically run your tests on their boxes on every push. It supports variety of platforms (Java, Python, Ruby, Node.Js, ..) and databases and other services (MySQL, PostgreSQL, Redis, Cassandra, RabbitMQ, …).

On top of that, it also offer a tight integration with Github. It supports features like automatic building of pull requests and updating of the commit status.

If you aren’t using it yet and your open-source project fits into one of supported platforms and use cases, you should really start using it.

Running your tests on a Raspberry Pi / ARM architecture

There are multiple different ways to get your test to run on a Raspberry Pi or a similar environment with an ARM architecture which resembles a Raspberry Pi set up.

Approach 1 - Running the tests directly on a Raspberry Pi

First and the most obvious one is to simply run the tests on the device itself. The problem with this approach is that is has multiple disadvantages such as low performance, lack of integration with Travis CI and a need to run and maintain your own test infrastructure.

Running the tests on the device itself would be ideal, but the lack of Travis CI integration makes it less attracting. It means I would need to some how get tests to automatically run on a Raspberry Pi and integrate the whole thing with Travis CI.

For the first task, I could potentially save myself some time by not writing my own system and simply use a Buildbot setup where master runs on a different server and a slave runs on a Raspberry Pi itself. And for all the people wondering, yes Jenkis is out of question because even a fine tuned JVM instance uses too much valuable memory and CPU which could otherwise be used by tests.

As you can see, setting up the integration might not be so bad, but maintaining the whole thing and making sure it’s running would be. I don’t have time to run and maintain yet another service and on top of that, Travis CI is much better and more effective at running a continuous integration environment than I am.

Approach 2 - Running the tests inside an emulated environment on Travis CI

Second approach is to emulate an ARM environment on i386 / x64 architecture. I decided to do just that. I used QEMU and chroot to emulate an ARM environment inside a Travis CI worker box.

Look ma, I have my tests running on Travis on ARM!

This approach has some downsides (more on that bellow), but it means that you don’t need to build and maintain a separate test infrastructure and your can reuse existing and familiar Travis CI work-flow.

The script

Bash script which sets up a chrooted QEMU ARM environment and runs your test can be found bellow.

#!/bin/bash
# Based on a test script from avsm/ocaml repo https://github.com/avsm/ocaml

CHROOT_DIR=/tmp/arm-chroot
MIRROR=http://archive.raspbian.org/raspbian
VERSION=wheezy
CHROOT_ARCH=armhf

# Debian package dependencies for the host
HOST_DEPENDENCIES="debootstrap qemu-user-static binfmt-support sbuild"

# Debian package dependencies for the chrooted environment
GUEST_DEPENDENCIES="build-essential git m4 sudo python"

# Command used to run the tests
TEST_COMMAND="make test"

function setup_arm_chroot {
    # Host dependencies
    sudo apt-get install -qq -y ${HOST_DEPENDENCIES}

    # Create chrooted environment
    sudo mkdir ${CHROOT_DIR}
    sudo debootstrap --foreign --no-check-gpg --include=fakeroot,build-essential \
        --arch=${CHROOT_ARCH} ${VERSION} ${CHROOT_DIR} ${MIRROR}
    sudo cp /usr/bin/qemu-arm-static ${CHROOT_DIR}/usr/bin/
    sudo chroot ${CHROOT_DIR} ./debootstrap/debootstrap --second-stage
    sudo sbuild-createchroot --arch=${CHROOT_ARCH} --foreign --setup-only \
        ${VERSION} ${CHROOT_DIR} ${MIRROR}

    # Create file with environment variables which will be used inside chrooted
    # environment
    echo "export ARCH=${ARCH}" > envvars.sh
    echo "export TRAVIS_BUILD_DIR=${TRAVIS_BUILD_DIR}" >> envvars.sh
    chmod a+x envvars.sh

    # Install dependencies inside chroot
    sudo chroot ${CHROOT_DIR} apt-get update
    sudo chroot ${CHROOT_DIR} apt-get --allow-unauthenticated install \
        -qq -y ${GUEST_DEPENDENCIES}

    # Create build dir and copy travis build files to our chroot environment
    sudo mkdir -p ${CHROOT_DIR}/${TRAVIS_BUILD_DIR}
    sudo rsync -av ${TRAVIS_BUILD_DIR}/ ${CHROOT_DIR}/${TRAVIS_BUILD_DIR}/

    # Indicate chroot environment has been set up
    sudo touch ${CHROOT_DIR}/.chroot_is_done

    # Call ourselves again which will cause tests to run
    sudo chroot ${CHROOT_DIR} bash -c "cd ${TRAVIS_BUILD_DIR} && ./.travis-ci.sh"
}

if [ -e "/.chroot_is_done" ]; then
  # We are inside ARM chroot
  echo "Running inside chrooted environment"

  . ./envvars.sh
else
  if [ "${ARCH}" = "arm" ]; then
    # ARM test run, need to set up chrooted environment first
    echo "Setting up chrooted ARM environment"
    setup_arm_chroot
  fi
fi

echo "Running tests"
echo "Environment: $(uname -a)"

${TEST_COMMAND}

And here is an example of a Travis CI config file which utilizes this script:

language: erlang

env:
  - ARCH=x64
  - ARCH=arm

script:
  - "bash -ex .travis-ci.sh"

As you can see, I have selected erlang for a language, but any less popular / used language on Travis CI should do.

Travis uses a dedicated set of VMs for each language and the reason for selecting a less popular one is so you don’t hog resources and further increase the wait time for other people who use a more popular language / runtime.

Downsides of the emulation approach

As noted above, this approach has some downsides. The biggest one is speed. You are emulating an ARM architecture on x86 without any hardware acceleration which slows things down. To make things even worse, Travis CI boxes are relatively small and they are already virtualized (inception!).

For the Luvit repository, whole ARM test run takes around ~40 minutes. That is pretty close to the 50 minutes hard time limit which Travis enforces. Out of those 40 minutes, about 10 are spent setting up a chrooted ARM environment and the rest of it is spent building the project and running the tests.

Originally, the build time took even longer, almost 55 minutes in fact, but I managed to “cheat” a bit and cut it down by 15 minutes or so. By default, Luvit comes bundled with libuv, zlib, yajl³, openssl and luajit dependency. Those dependencies are then built when building the whole project. To speed things up, I indicated to the build system to use an existing system libraries instead of building all the bundled dependencies.

Doing that for yajl, zlib and some other dependencies is not a big issue, since those dependencies are rarely updated and Ubuntu 12.04 repositories already includes a fairly recent version of those libraries.

There are a couple of exceptions though. The most notable ones are libuv and openssl. Libuv dependency is updated fairly frequently and not available in the standard Ubuntu repositories. And as far as the OpenSSL goes - some of the code and tests rely on a new functionality which is not available in the older versions of OpenSSL which is typically available in your distribution package repositories.

It’s also good to add that the actual test run itself is pretty fast (couple of minutes) and as noted above, most of the time is spent on building the project.

Conclusion

Pretty much every decision you make in life (computer science related or not) is about making some kind of a trade-off and this one is no different. The solution I have described is not ideal, but relatively slow automated tests are in many cases⁴ better than no automated tests :-)

environment more similar to other production environments, they have, not so long ago, migrated their boxes to x64.

that’s out of scope of this post.

runs are so slow so they discourage people from contributing.