Functional Testing and YUI

For the past month or so, I’ve had the job of creating regression
functional tests for some code running YUI3. The goal of these tests is
to help build a basic regression suite so that when upgrading YUI, we
can have a quick sanity check to see if anything broke.

YUI Test is a pretty solid test library for JavaScript, but there are
some shortcomings. Specifically, YUI Test seems more focused on ‘unit’
testing versus ‘functional’ testing. Actually, most test libraries out
there are (CasperJs and Selenium being exceptions).

In tandem with YUI Test, I’ve also been giving YETI a try. The goal of
all this is to have an easy way to automate a large-scale regression of
tens of components (and hopefully hundreds one day).

I want to capture some of the difficulties I’ve faced along the way:

1. Detecting where focus works when executing a test inside a browser,
   but it fails when running it through Yeti. The reason is that some
   browsers will say a page is unfocused if it’s being run in the
   background. This sort of makes sense, but it makes it very difficult
   to test whether a link has focus, because if the page doesn’t have
   focus, the link certainly won’t. So you have a case where when run
   individually outside of Yeti, the tests pass, but when run in Yeti
   they fail. Not so great.
2. There doesn’t seem to be any functionality for building out template
   HTML on each test execution. This is useful for when you have some
   HTML that’s needed in order for a widget to be set up, but you don’t
   want your tests fighting with each other (e.g. a test leaving a
   widget in a non-default state). I went ahead and wrote a quick
   utility that does this set up and tear down and hope to commit it
   back.
3. YUI Test doesn’t have a built-in way of determining if something is
   visible or the right dimensions. Again, I went ahead and wrote a
   utility that does this and hope to commit it back.
4. Page navigation is impossible. But I think that’s going to be an
   issue with any JS test framework that can’t drive a browser.
5. Simulating keyboard events isn’t perfect. I wanted to test what
   happens when the user presses tab at the bottom of a modal, but when
   simulating it, the browser basically ignores the event and doesn’t
   move focus like it would if the user pressed the tab key. Again, i
   think you need a browser driver to do this type of stuff.

I have looked into CasperJS and Selenium, but each have their cons:

• CasperJS is limited to testing a headless WebKit browser, so the
  scope of the tests isn’t that great. Considering most things break
  in IE, not testing in IE doesn’t seem like a good regression suite.
• Selenium is much slower to execute than I’d like. Launching the
  browsers take a substantial amount of time when you’re trying to
  tweak a test. That being said, I may look into Selenium more, now
  that I know about WebDriver. Plus,
  maybe I’m overplaying the slowness. Perhaps it’s fine to be a little
  slow, esp. since these functional tests won’t be run too frequently
  during normal development.