For the past month or so, I’ve had the job of creating regression
functional tests for some code running YUI3. The goal of these tests is
to help build a basic regression suite so that when upgrading YUI, we
can have a quick sanity check to see if anything broke.
some shortcomings. Specifically, YUI Test seems more focused on ‘unit’
testing versus ‘functional’ testing. Actually, most test libraries out
there are (CasperJs and Selenium being exceptions).
In tandem with YUI Test, I’ve also been giving YETI a try. The goal of
all this is to have an easy way to automate a large-scale regression of
tens of components (and hopefully hundreds one day).
I want to capture some of the difficulties I’ve faced along the way:
- Detecting where focus works when executing a test inside a browser,
but it fails when running it through Yeti. The reason is that some
browsers will say a page is unfocused if it’s being run in the
background. This sort of makes sense, but it makes it very difficult
to test whether a link has focus, because if the page doesn’t have
focus, the link certainly won’t. So you have a case where when run
individually outside of Yeti, the tests pass, but when run in Yeti
they fail. Not so great.
- There doesn’t seem to be any functionality for building out template
HTML on each test execution. This is useful for when you have some
HTML that’s needed in order for a widget to be set up, but you don’t
want your tests fighting with each other (e.g. a test leaving a
widget in a non-default state). I went ahead and wrote a quick
utility that does this set up and tear down and hope to commit it
- YUI Test doesn’t have a built-in way of determining if something is
visible or the right dimensions. Again, I went ahead and wrote a
utility that does this and hope to commit it back.
- Page navigation is impossible. But I think that’s going to be an
issue with any JS test framework that can’t drive a browser.
- Simulating keyboard events isn’t perfect. I wanted to test what
happens when the user presses tab at the bottom of a modal, but when
simulating it, the browser basically ignores the event and doesn’t
move focus like it would if the user pressed the tab key. Again, i
think you need a browser driver to do this type of stuff.
I have looked into CasperJS and Selenium, but each have their cons:
- CasperJS is limited to testing a headless WebKit browser, so the
scope of the tests isn’t that great. Considering most things break
in IE, not testing in IE doesn’t seem like a good regression suite.
- Selenium is much slower to execute than I’d like. Launching the
browsers take a substantial amount of time when you’re trying to
tweak a test. That being said, I may look into Selenium more, now
that I know about WebDriver. Plus,
maybe I’m overplaying the slowness. Perhaps it’s fine to be a little
slow, esp. since these functional tests won’t be run too frequently
during normal development.