Caching Dependencies and Directories

The features described here are currently only available for private repositories on travis-ci.com and our new container-based infrastructure.

These features are also still experimental, please contact us with any questions, issues and feedback.

Cache content can be accessed by pull requests

Do note that cache content will be available to any build on the repository, including Pull Requests. Do exercise caution not to put any sensitive information in the cache, lest a malicious attacker potentially expose it.

Caching directories (Bundler, dependencies)

With caches, Travis CI can persist directories between builds. This is especially useful for dependencies that need to be downloaded and/or compiled from source.

Build phases

Travis CI attempts to upload cache after script, but before either after_success or after_failure is run. Note that the failure to upload the cache does not mark the job a failure.

Bundler

On Ruby and Objective-C projects, installing dependencies via Bundler can make up a large portion of the build duration. Caching the bundle between builds drastically reduces the time a build takes to run.

The logic for fetching and storing the cache is described below.

Enabling Bundler caching

Bundler caching is automatically enabled for Ruby projects that include a Gemfile.lock.

(Bundler caching is not yet enabled automatically)

You can explicitly enable Bundler caching in your .travis.yml:

language: ruby
cache: bundler

Whenever you update your bundle, Travis CI will also update the cache.

Determining the bundle path

Travis CI tries its best at determining the path bundler uses for storing dependencies.

If you have custom Bundler arguments, and these include the –path option, Travis CI will use that path. If –path is missing but –deployment is present, it will use vendor/bundle.

Otherwise it will automatically add the –path option. In this case it will either use the value of the environment variable BUNDLE_PATH or, if it is missing, vendor/bundle.

CocoaPods

On Objective-C projects, installing dependencies via CocoaPods can take up a good portion of your build. Caching the compiled Pods between builds helps reduce this time.

Enabling CocoaPods caching

You can enable CocoaPods caching for your repository by adding this to your .travis.yml:

language: objective-c
cache: cocoapods

If you want to enable both Bundler caching and CocoaPods caching, you can list them both:

language: objective-c
cache:
  - bundler
  - cocoapods

Note that CocoaPods caching won’t have any effect if you are already vendoring the Pods directory in your Git repository.

Determining the Podfile path

By default, Travis CI will assume that your Podfile is in the root of the repository. If this is not the case, you can specify where the Podfile is like this:

language: objective-c
podfile: path/to/Podfile

pip cache

For caching pip files, use:

language: python

cache: pip

caches $HOME/.cache/pip.

ccache cache

For caching ccache files, use:

language: c # or other C/C++ variants

cache: ccache

caches $HOME/.ccache, and adds /usr/lib/ccache to the front of $PATH.

R package cache

For caching R packages, use:

language: R

cache: packages

This caches $HOME/R/Library, and sets R_LIB_USER=$HOME/R/Library environment variable.

Rust Cargo cache

For caching Cargo packages, use:

language: rust

cache: cargo

This caches $HOME/.cargo and $TRAVIS_BUILD_DIR/target.

Arbitrary directories

You can cache arbitrary directories, such as Gradle, Maven, Composer and npm cache directories, between builds by listing them in your .travis.yml:

cache:
  directories:
  - .autoconf
  - $HOME/.m2

As you can see, it is also possible to use environment variables in the directories.

The logic for fetching and storing the cache is described below.

Things not to cache

The cache’s purpose is to make installing language-specific dependencies easy and fast, so everything related to tools like Bundler, pip, Composer, npm, Gradle, Maven, is what should go into the cache.

For other things, the cache won’t be an improvement. Installing them usually takes only short amounts of time, but downloading them will be the same speed when pulled from the cache as it will be from their original source. You possibly won’t see a speedup putting them into the cache.

Things like:

  • Android SDKs
  • Debian packages
  • JDK packages
  • Compiled binaries

Anything that’s commonly not changing is better suited for something like our APT caching proxy. Please shoot us an email and we’ll see about adding your custom source to our cache.

Fetching and storing caches

  • Travis CI fetches the cache for every build, including feature branches and pull requests.
  • There is one cache per branch and language version/ compiler version/ JDK version/ Gemfile location/ etc.
  • Pull requests use the cache of the target of the pull request.
  • If a branch does not have its own cache yet, it uses the master branch cache (unless it is a pull request, see above).
  • Only modifications made to the cached directories from normal pushes are stored.

before_cache phase

When using caches, it may be useful to run command just prior to uploading the new cache archive. For example, the dependency management utility may write log files into the directory you are watching, and you would do well to ignore these.

For this purpose, you can use before_cache phase.

cache:
  directories:
    - $HOME/.cache/pip
⋮
before_cache:
  - rm -f $HOME/.cache/pip/log/debug.log

Failures in this stage does not mark the job a failure.

Clearing Caches

Sometimes you spoil your cache by storing bad data in one of the cached directories.

Caches can also become invalid if language runtimes change and the cache contains native extensions. (This often manifests as segmentation faults.)

You can access caches in one of the two ways. Each method also gives you a means of deleting caches.

  1. On the web https://travis-ci.com/OWNER/REPOSITORY/caches for private repositories or https://travis-ci.org/OWNER/REPOSITORY/caches for public repositories, which is accessible from the Settings menu

    Image of cache UI

  2. With command line client:

travis cache --delete

Running travis cache --delete inside the project directory.

There is also a corresponding API for clearing the cache.

Configuration

Enabling multiple caching features

When you want to enable multiple caching features, you can list them as an array:

cache:
- bundler
- apt

This does not when caching arbitrary directories. If you want to combine that with other caching modes, you will have to use a hash map:

cache:
  bundler: true
  directories:
  - vendor/something
  - .autoconf

Explicitly disabling caching

You can explicitly disable all caching by setting the cache option to false in your .travis.yml:

cache: false

It is also possible to disable a single caching mode:

cache:
  bundler: false
  apt: true

Setting the timeout

Caching has a timeout set to 5 minutes by default. The timeout is there in order to guard against any issues that may result in a stuck build. Such issues may be caused by a network issue between worker servers and S3 or even by a cache being to big to pack it and upload it in timely fashion. There are, however, situations when you might want to set a bigger timeout, especially if you need to cache large amount. In order to change the timeout you can use the timeout property with a desired time in seconds:

cache:
  timeout: 1000

How does the caching work?

The caching tars up all the directories listed in the configuration and uploads them to S3, using a secure and protected URL, ensuring security and privacy of the uploaded archives.

Note that this makes our cache not network-local, it’s still bound to network bandwidth and DNS resolutions for S3. That impacts what you can and should store in the cache. If you store archives larger than a few hundred megabytes in the cache, it’s unlikely that you’ll see a big speed improvement.

Before the build, we check if a cached archive exists. If it does, we pull it down and unpack it to the specified locations.

After the build we check for changes in the directory, create a new archive and upload the updated archive back to S3.

The upload is currently part of the build cycle, but we’re looking into improving that to happen outside of the build, giving faster build feedback.