Functional Testing and YUI

For the past month or so, I’ve had the job of creating regression
functional tests for some code running YUI3. The goal of these tests is
to help build a basic regression suite so that when upgrading YUI, we
can have a quick sanity check to see if anything broke.

YUI Test is a pretty solid test library for JavaScript, but there are
some shortcomings. Specifically, YUI Test seems more focused on ‘unit’
testing versus ‘functional’ testing. Actually, most test libraries out
there are (CasperJs and Selenium being exceptions).

In tandem with YUI Test, I’ve also been giving YETI a try. The goal of
all this is to have an easy way to automate a large-scale regression of
tens of components (and hopefully hundreds one day).

I want to capture some of the difficulties I’ve faced along the way:

  1. Detecting where focus works when executing a test inside a browser,
    but it fails when running it through Yeti. The reason is that some
    browsers will say a page is unfocused if it’s being run in the
    background. This sort of makes sense, but it makes it very difficult
    to test whether a link has focus, because if the page doesn’t have
    focus, the link certainly won’t. So you have a case where when run
    individually outside of Yeti, the tests pass, but when run in Yeti
    they fail. Not so great.
  2. There doesn’t seem to be any functionality for building out template
    HTML on each test execution. This is useful for when you have some
    HTML that’s needed in order for a widget to be set up, but you don’t
    want your tests fighting with each other (e.g. a test leaving a
    widget in a non-default state). I went ahead and wrote a quick
    utility that does this set up and tear down and hope to commit it
  3. YUI Test doesn’t have a built-in way of determining if something is
    visible or the right dimensions. Again, I went ahead and wrote a
    utility that does this and hope to commit it back.
  4. Page navigation is impossible. But I think that’s going to be an
    issue with any JS test framework that can’t drive a browser.
  5. Simulating keyboard events isn’t perfect. I wanted to test what
    happens when the user presses tab at the bottom of a modal, but when
    simulating it, the browser basically ignores the event and doesn’t
    move focus like it would if the user pressed the tab key. Again, i
    think you need a browser driver to do this type of stuff.

I have looked into CasperJS and Selenium, but each have their cons:

  • CasperJS is limited to testing a headless WebKit browser, so the
    scope of the tests isn’t that great. Considering most things break
    in IE, not testing in IE doesn’t seem like a good regression suite.
  • Selenium is much slower to execute than I’d like. Launching the
    browsers take a substantial amount of time when you’re trying to
    tweak a test. That being said, I may look into Selenium more, now
    that I know about WebDriver. Plus,
    maybe I’m overplaying the slowness. Perhaps it’s fine to be a little
    slow, esp. since these functional tests won’t be run too frequently
    during normal development.