The Lamp Post

A blog about life, productivity and front-end web development.

mail twitter rss2 youtube twitch steam2 github linkedin stackoverflow
The Web App Testing Guidebook - UI Testing of Real World Websites Using WebdriverIO

Front-end Summit 2015 - This Tuesday

Save to Pocket

This Tuesday I have the pleasure of presenting at Front-End Summit online (sign up with discount code KEVIN for 20% off).

The topic will be Visual Regression Testing, something I'm try to figure out. I say "trying to figure out", because I keep making progress on a specific toolset for it, then I see a better looking one come out and start all over.

Well, that's what I'll be talking about really. In my experience, it's tricky to get started with visual regression testing (but oh so worth it in my opinion). Not only because of the changing tool landscape, but also because it's still a fairly new test discipline.

I don't have time to get in to too many details, but here's the outline of the presentation (also available on Workflowy which will be more up-to-date):

  • Story about messing up on a small update
    • Tasked with making a small fix to the search results
    • Made the fix, tested in all the right environments
    • Caught a regression on a page I didn't know about
    • Embarrassment of not validating
    • Reasons all pages won't be checked:
      • Laziness & Ignorance
      • I'm either too new to the project to know where to check, or i'm cutting corners due to test fatigue
      • It's too big of a website to know where to test everything
  • What is VRT?
    • Visual
      • Screenshot based, not unit tests
      • Test by comparing the visual output (i.e. diffing) between two versions.
    • Regressions
      • What are regressions?
      • Not testing new stuff, validating that old things still work as expected
    • Testing
      • “Testing shows the presence, not the absence of bugs.” -Edsger W. Dijkstra
      • Not perfect
      • Won't catch everything
  • Difficulties
    • Websites are incredibly complex
      • Personas
        • Dynamic site w/Angular/React/etc
        • Simple site you haven't touched in a year
        • 1000+ page CMS
        • Visual/interactive page with animations, background image stuff, etc
        • Responsive layouts with content that's different per screen resolution even though it's the same page
        • Brand new sites under heavy development
        • A/B Sites
        • Mobile web apps
        • Sites behind logins
          • Internal portal with SSO
        • Sites with server/API functionality
          • Responses from the server
        • Content that changes automatically on the page (timer/carousel/etc)
        • Content that loads on scroll
        • "Front" pages with content that changes daily
          • Twitter or a news page
      • No "one-size-fits-all" approach
    • Different content between dev and staging
    • CSS animations with long delays
    • There are lots of tools out there
      • While they're all good at something, they all suck at something
  • Benefits
    • Avoid style drift
    • It's a built in "smoke" test
      • Has saved me before simply by validating that a page loads the data right
  • Things VRT can't do (right now and as far as I know)
    • Ensure style consistency/governance across pages
      • Maybe unit tests can help with this
    • Compare differences between browsers
    • Work magic
  • Making the VRT Plan
    • often the thought seems overwhelming
      • Take a single page
        • Break it down
        • Rule out complex stuff
          • Get fine-tuned where necessary
        • Group items
          • Write test suites based on components. Piece together to test pages
    • Where to store your baseline images?
    • Style-guide based testing
    • Fake data based testing
    • Build system integration
    • Review process/team flow
    • discriminate by priority, complexity, easibility, rate of change.
  • Toolsets
    • PhantomCSS
    • Gemini
    • WebdriverCSS
    • Wraith
    • DiffUX
    • Dpxdt
    • Shoov
  • Selling VRT
    • Get someone "on your team" with it
    • VRT has 'functional' testing built in
      • It might save you time from building out a completely separate functional test suite
    • Reduces outages
    • Reduces client embarrassment
  • Final Thoughts
    • It's difficult - so make it fun
      • You're teaching a computer how to use your website, that's pretty neat
      • What's your hook? At what point does VRT become a positive feedback loop
        • For even ounce of effort you put in, you get 2 ounces of worth back
      • Team up
        • You're more likely to be successful if you have someone working with you (and just as passionate about it).
    • It's difficult - Plan for success
      • Take baby steps
      • Play the long game
      • Don't allocate too much time in one sitting
        • Break up the work to form better habits
    • You won't have perfect tests, but you will have valuable tests
  • Learning More