Note: It's been a while since I originally wrote this. Nowadays I use WebdriverIO and WebdriverCSS. If you're interested in learning those tools, I've got a free short course available at http://learn.visualregressiontesting.com
Over the past four years, I've learned the hard way about how painful it can be to try and update a codebase used by a large number of applications. Changes that seem innocent can break a specific use case that wasn't anticipated. Do enough manual regression testing and you will catch the bugs, but it's' costly and time consuming process. When things are costly and time consuming, they usually stop being done.
This is why I focus a lot of my efforts towards build process automation. Computers are fantastic about doing boring, repetitive work and never complaining. If you can get them to do the boring chores for you, you spend more time doing the fun, challenging work.
UI regression testing is one of those spots where I'm looking for automation scripts to take over. Not only because playing "spot the difference" between your builds is boring, but also because we're horrible at it.
Wraith
There are several UI regression tools currently available and in my opinion Wraith takes the cake for ease of use. Setup is fairly simple, as no database or server is needed to run the scripts. It also generates a nice gallery of before and after shots, along with a diff of the two.
Wraith is missing some features. As far as I know, if you have any dynamic actions on your page (e.g. tabs), you have to implement your own way to access those actions (so that a screenshot can be taken of that particular view). Also, since its default is to capture a screenshot of the entire page, any vertical alignment change at the top of the page can cascade down to everything below it.
That said, it's a great solution to getting started with UI regression testing.
Installation
Thanks to ruby gems, installation is pretty simple. The line to install Wraith is just:
gem install wraith
I didn't have to set up any specific Ruby environment information inside of .travis.yml; it works straight out of the box with the minimal Ruby installation that all TravisCI environments have.
Wraith Configuration
Most of the defaults were left in place for the Wraith config file. I removed the extra screen width configs, because our framework currently doesn't use media queries.
Defining What to Compare
Wraith works by comparing two different sites against each other, which is perfect for a "dev vs. prod" comparison. In our config, we compare a local server that TravisCI spins up against our production GitHub Pages documentation site.
Fortunately, our documentation site doesn't require a login or VPN access. This keeps the setup pretty simple. If we did need a login, Wraith provides some customization with cookies and HTTP Headers and such, so it would be possible to configure this if needed.
Building the Component Paths
Since we're using AngularJS, the built-in indexing tool for Wraith didn't work out of the box. Rather than spend time getting it to work, I wrote a simple config for our 'grunt-replace' task that will create the sitemap based on an array of components that gets built during our build process. I admit it's not the smoothest of methods, but it works for now.
It's worth noting that I originally wanted to leave the component name out of the page path, so:
paths:
- "/#/component/configs"
- "/#/component/hotkeys"
However, this caused image path issues in the gallery, due to the URL containing #. I tried patching the gallery code up, but realized after some effort (too much really) that all I needed to do was name the paths (e.g. configs: "/#/component/configs") and the gallery code would use the name instead of a path.
Reducing Noise
Wraith allows you to configure how screenshots are shown in the generated gallery. For our needs, we only wanted to see the pages that had a diff (to help keep the review process fast). To do that, we added mode: diffs_only to our config.yaml file.
Running Wraith via grunt
Sometimes we'd like to run our diff locally, before pushing a PR to Travis. To do this, I added a config to our grunt-shell task:
wraith: {
command: 'wraith capture config',
options: {
stdout: true,
execOptions: {
cwd: '<%= config.wraith %>'
}
}
}
In this config, I tell grunt-shell to run the Wraith CLI command from the Wraith folder (config.wraith is defined in our grunt config.js file).
I really like grunt-shell for integrating CLI commands into our Grunt processes. Instead of having to remember several specific CLI commands, I run grunt wraith and it does it all for me. It also means that any updates to the Wraith setup in the future don't require re-learning the commands (especially useful for non-core contributors).
Integrating with TravisCI
Travis mirrors our local setup, so most of what we do is just run our commands.
Running only on successful build
Since Travis starts with a clean box on every build, it requires installing Wraith on every run. We only want it to do that after we're sure the code passes all the other build steps. This way we avoid installing Wraith on a broken codebase (i.e. get to the code failures before worrying about the UI failures).
.travis.yml allows you to execute code after_success. Anything defined in this section will run after the build process has successfully completed. Here's an example:
script:
# build and test the code
- grunt
- grunt test:full
after_success:
# install and run wraith after the code is successfully built
- gem install wraith && grunt wraith'
If the build fails, all the Wraith steps are skipped, saving us time.
Running only for PRs
While we could run Wraith for every commit, it doesn't make sense to view a UI regression for code that isn't ready. If you want to see a UI regression for your branch, you should run it locally from your own computer.
TravisCI exposes environmental variables, one of which is the Pull Request number. We can use this to execute code only for PRs:
after_success:
# install and run wraith after the code is successfully built (and only on a PR)
- '[ "${TRAVIS_PULL_REQUEST}" != "false" ] && gem install wraith && grunt wraith || false'
If the build isn't a PR, the variable is set to false, which fails our conditional.
Note: It's a little ugly to string together commands like this (I prefer individual commands on separate lines), but in this instance it's prettier than repeating the conditional:
- '[ "${TRAVIS_PULL_REQUEST}" != "false" ] && gem install wraith || false'
- '[ "${TRAVIS_PULL_REQUEST}" != "false" ] && grunt wraith || false'
There might be a better way to avoid this, but I'm still a novice to the yaml format and shell scripts. Suggestions welcome!
Deploying to a CDN
Once the Wraith gallery has been built, we need to store it outside of the temporary Travis box. There is a deploy cycle built in to Travis, but it doesn't trigger for PR builds. While Travis offers other options such as Artifacts, I found using the grunt-cloudfiles task to be the simplest solution.
wraith: {
'user': 'encorecloudfiles',
'key': process.env.cloudFilesApi,
'region': 'IAD',
'upload': [{
'container': 'encore-ui-wraith',
'src': '<%= config.wraith %>/shots/**/*',
'dest': '/<%= grunt.option("pr") %>/',
'stripcomponents': 2
}]
}
Deploying to the Rackspace Cloud Files API requires an API key, which we provide as an encrypted value inside our Travis.yml file.
To generate our secure token, we use the Travis CLI:
travis encrypt cloudFilesApi=secretvalue
As said, we store the value in our travis.yml file. At runtime, Travis will decrypt the key and provide it as an environment variable. This allows us to securely store the key on our public repo, but still access it in a decrypted form.
Alternatively, Travis now supports adding encrypted variables via their site.
With the cloudfiles task set up, we can add it to our after_success tasks:
- '[ "${TRAVIS_PULL_REQUEST}" != "false" ] && gem install wraith && grunt wraith && grunt cloudfiles:wraith --pr ${TRAVIS_PULL_REQUEST} || false'
Keeping PR Builds Separate
If you didn't notice in the task config, the Wraith files are deployed to a folder specific to that Pull Request. We use the PR number to name the folder, so that the results from one PR don't overwrite other PRs open at the same time.
To get the PR number into Grunt, we pass it in as a CLI parameter. We can access any CLI parameter using grunt.option("nameOfParam"). For example, running grunt cloudfiles:wraith --pr 42 will result in grunt.option("pr") returning 42.
Adding a Gallery Link Comment to the PR
It would be tedious to try and remember the link to our CDN each time we want to view the gallery. To avoid this, we have a dummy GitHub account post a comment to the PR with the link. The process is similar to what we've done before, except it doesn't use Grunt.
We run the gh-pr-wraith.js nodejs shell script, passing in the PR number again as a parameter. This will make a call to the GitHub API using the token for the dummy GitHub account, with a comment linking to the generated Wraith page. That comment will be posted to the GitHub PR review page, for easy access from a reviewer standpoint.
Here's the full after_success code with all our steps:
- '[ "${TRAVIS_PULL_REQUEST}" != "false" ] && gem install wraith && grunt wraith || false'
- '[ "${TRAVIS_PULL_REQUEST}" != "false" ] && grunt cloudfiles:wraith --pr ${TRAVIS_PULL_REQUEST} && node utils/gh-pr-wraith.js ${TRAVIS_PULL_REQUEST} || false'
We split it on to two lines for readability. I'd like to have this whole thing in a single script, but that's work for another day.
Summary
It was a little bit of effort, but this automation step provides value to the project by making PR reviews just a little bit easier. Instead of requiring a reviewer to check out the branch and build the code, then try and remember what the previous version looked like, we can quickly view a gallery of image diffs between Prod and the PR code.