GUI and Headless Browser Testing

This guide covers headless GUI & browser testing using tools provided by the Travis CI environment. Most of the content is technology-neutral and does not cover all the details of specific testing tools (like Poltergeist or Capybara). We recommend you start with the Onboarding and Build Configuration guides before reading this one.

Work with Sauce Labs #

Sauce Labs provides a Selenium cloud with access to more than 170 different device/OS/browser combinations. If you have browser tests that use Selenium, using Sauce Labs to run the tests is very easy. First, you need to sign up for their service (it’s free for open-source projects).

Once you’ve signed up, set up a tunnel using Sauce Connect so Sauce Labs can connect to your web server. Our Sauce Connect addon makes this easy, just add this to your .travis.yml:

addons:
  sauce_connect:
    username: "Your Sauce Labs username"
    access_key: "Your Sauce Labs access key"

You can encrypt your access key, if you want to.

Now, Sauce Labs has a way of reaching your web server, but you still need to start it up. See Starting a Web Server below for more information on how to do that.

Finally, you need to configure your Selenium tests to run on Sauce Labs instead of locally. This is done using a Remote WebDriver. The exact code depends on what tool/platform you’re using, but for Python, it would look like this:

username = os.environ["SAUCE_USERNAME"]
access_key = os.environ["SAUCE_ACCESS_KEY"]
capabilities["tunnel-identifier"] = os.environ["TRAVIS_JOB_NUMBER"]
hub_url = "%s:%s@localhost:4445" % (username, access_key)
driver = webdriver.Remote(desired_capabilities=capabilities, command_executor="http://%s/wd/hub" % hub_url)

The Sauce Connect addon exports the SAUCE_USERNAME and SAUCE_ACCESS_KEY environment variables, and relays connections to the hub URL back to Sauce Labs.

This is all you need to get your Selenium tests running on Sauce Labs. However, you may want to only use Sauce Labs for Travis CI builds, and not for local builds. To do this, you can use the CI or TRAVIS environment variables to conditionally change what driver you’re using (see our list of available environment variables for more ways to detect if you’re running on Travis CI).

To make the test results on Sauce Labs a little more easy to navigate, you may wish to provide some more metadata to send with the build. You can do this by passing in more desired capabilities:

capabilities["build"] = os.environ["TRAVIS_BUILD_NUMBER"]
capabilities["tags"] = [os.environ["TRAVIS_PYTHON_VERSION"], "CI"]

For travis-web, our very own website, we use Sauce Labs to run browser tests on multiple browsers. We use environment variables in our .travis.yml to split up the build into multiple jobs, and then pass the desired browser into Sauce Labs using desired capabilities. On the Travis CI side, it ends up looking like this.

Run GUI Tests with xvfb #

To run tests requiring a graphical user interface on Travis CI, use xvfb (X Virtual Framebuffer) to imitate a display. If you need a browser, you can use Firefox (either with the pre-installed version, or the addon) or Google Chrome (with the addon, on Linux Trusty or macOS).

Use services: on your script #

This only works on Ubuntu 16.04 (Xenial) and later on releases i.e. with dist: xenial or dist: bionic

The following will start xvfb and set the right values for the DISPLAY environment variable:

dist: xenial
services:
  - xvfb

How to use the xvfb-run wrapper #

xvfb-run is a wrapper for invoking xvfb so that xvfb can be used with less fuss:

script: xvfb-run make test

To set the screen resolution:

script: xvfb-run --server-args="-screen 0 1024x768x24" make test

Direct use of xvfb #

This is recommended on Ubuntu 14.04 (Trusty) i.e. with dist: trusty. For dist: xenial, use the services keyword described above.

To use xvfb itself, start it in the before_script section of your .travis.yml:

before_script:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
  - sleep 3 # give xvfb some time to start

Note: Don’t run xvfb directly, as it does not handle multiple concurrent instances that way.

If you need to set the screen size and pixel depth, you need to start xvfb with the start-stop-daemon utility and not with the init script in the previous example.

For example, to set the screen resolution to 1280x1024x16:

before_install:
  - "/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16"

See xvfb manual page for more information.

Start a Web Server #

If your project requires a web application running to be tested, you need to start one before running tests. It is common to use Ruby, Node.js and JVM-based web servers that serve HTML pages used to run test suites. Because every build environment provides at least one version of Ruby, Node.js and OpenJDK, you can rely on one of those three options.

Add a before_script to start a server, for example:

before_script:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
  - sleep 3 # give xvfb some time to start
  - rackup  # start a Web server
  - sleep 3 # give Web server some time to bind to sockets, etc

If you need web server to be listening on port 80, remember to use sudo (Linux will not allow non-privileged process to bind to port 80). For ports greater than 1024, using sudo is not necessary (and not recommended).

Headless mode with the Chrome addon #

Starting with version 57 for Linux Trusty and version 59 on macOS, Google Chrome can be used in “headless” mode with the Chrome addon, which is suitable for driving browser-based tests using Selenium and other tools.

As of 2017-05-02, this means stable or beta on Linux builds, and beta on macOS builds.

For example, on Linux

dist: xenial
addons:
  chrome: stable
before_install:
  - # start your web application and listen on `localhost`
  - google-chrome-stable --headless --disable-gpu --remote-debugging-port=9222 http://localhost &
  ⋮

On macOS:

language: objective-c
addons:
  chrome: beta
before_install:
  - # start your web application and listen on `localhost`
  - "/Applications/Google\\ Chrome.app/Contents/MacOS/Google\\ Chrome --headless --disable-gpu --remote-debugging-port=9222 http://localhost &"
  ⋮

More Documentation #

• Headless Chromium documentation
• Getting Started with Headless Chrome

Headless mode with the Firefox addon #

Starting with version 56, Firefox can be used in “headless” mode with the Firefox addon, which is suitable for driving browser-based tests using Selenium and other tools. Headless mode can be enabled using the MOZ_HEADLESS environment variable:

env:
  global:
    - MOZ_HEADLESS=1
addons:
  firefox: latest

Alternatively, you can pass the -headless command line argument when starting Firefox. For example, the following code demonstrates how you would set this argument using the Python client for Selenium:

from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument('-headless')
firefox = Firefox(firefox_options=options)

More Documentation #

• Using headless mode
• Automated testing with headless mode

Use PhantomJS #

PhantomJS is a headless WebKit with JavaScript API. It is an optimal solution for fast headless testing, site scraping, pages capture, SVG renderer, network monitoring and many other use cases.

CI environment provides PhantomJS pre-installed (available in PATH as phantomjs; don’t rely on the exact location). Since it is completely headless, there is no need run xvfb.

A very simple example:

script: phantomjs testrunner.js

If you need a web server to serve the tests, see the previous section.

Examples #

The following are a series of examples.

Real World Projects #

• Ember.js (starts web server programmatically)
• Sproutcore (starts web server with before_script)

Ruby Example #

RSpec, Jasmine, Cucumber #

Here’s an example rake task that runs Rspec, Jasmine, and Cucumber tests:

task :travis do
  ["rspec spec", "rake jasmine:ci", "rake cucumber"].each do |cmd|
    puts "Starting to run #{cmd}..."
    system("export DISPLAY=:99.0 && bundle exec #{cmd}")
    raise "#{cmd} failed!" unless $?.exitstatus == 0
  end
end

In this example, both Jasmine and Cucumber need the display port, because they both use real browsers. Rspec would run without it, but it does no harm to set it.

Troubleshooting #

Selenium and Firefox popups #

If your test suite handles a modal dialog popup, for example, a redirect to another location, then you may need to add a custom profile so that the popup is suppressed.

This can be fixed by applying a custom Firefox profile with the option turned off: (example is in Ruby using Capybara)

Capybara.register_driver :selenium do |app|

  custom_profile = Selenium::WebDriver::Firefox::Profile.new

  # Turn off the super annoying popup!
  custom_profile["network.http.prompt-temp-redirect"] = false

  Capybara::Selenium::Driver.new(app, :browser => :firefox, :profile => custom_profile)
end

Karma and Firefox inactivity timeouts #

When testing with Karma and Firefox, you may encounter build errors as a result of browser inactivity timeouts. When this occurs, Karma will output an error similar to:

WARN [Firefox 31.0.0 (Linux)]: Disconnected (1 times), because no message in 10000 ms.

In that case, you should increase the browser inactivity timeout to a higher value in karma.conf.js, e.g.:

browserNoActivityTimeout: 30000,

For more information, refer to the Karma Configuration File documentation.