⚡️ Robo-Testing your Website Without Writing Code ️🤖
Getting Started with UI Test Automation
This is the first part of a series of posts which will eventually comprise a “Getting Started with UI Test Automation” mega-post, drawing from 2 years worth of growing our automated testing suite at Code.org. I didn’t know the first thing about automated testing when we started out, so I’m writing the notes I wish I had when I was getting started.
In this post you will learn why it’s often worth automating your otherwise manual dev/production build verification tests, how to choose between “little to no code” and “code-heavy” options, and—for “codeless” tests—what services are worth taking a look at & why.
While I’m going to give my personal opinion on these services in this series, I’d like to disclaim that in my work at non-profit Code.org we use Applitools Eyes and Sauce Labs extensively, and they are two of our generous infrastructure partners (thanks for helping bring not-broken CS curricula to classrooms!).
To code, or not to code?
Rolling your own test suite confers a number of advantages over time, but it’s not always the right fit for your project. A single startup founder hoping to get a handle on egregious bugs on deploys / time wasted manual testing might find spinning up a Selenium Grid a bit of a time sink, and conversely a growing team might find simpler testing services constraining or increasingly expensive / slow to verify builds as their test suite and application complexity grows.
Today’s Option A: Tests without Code
My long-time friend Matt, CEO of a neat real-time student rewarding, reporting and data website aptly named LiveSchool, was asking about lightweight testing options for a new front-end after seeing my Tweets about our testing suite at Code.org.
I gave him a high-level overview, sharing our experience growing our stack and the tools and services we started using to keep (1) our team from bumping elbows and (2) our code from ruining a classroom’s planned lessons. I painstakingly detailed when to choose cucumber over raw Ruby, whether to use selectors or visual testing, and how to spin up browsers in the cloud.
But Matt wanted something simple, and knew I was holding out on him. He had used WebFlow to transform the web design and prototyping workflow at LiveSchool — there must, he thought, be something similar.
OK, now what are the cheap and easy options you alluded to? I know you know!
(He knows me well—I was holding back.)
Automated testing isn’t just for our dedicated, hardworking, Java-dependency-wrangling automated QA professional friends (💘) anymore. There now exists a healthy set of web services that can be used to spin up and trigger automated, visual testing—all without leaving your browser.
Who is non-code web-service-based automated testing best for?
- Startup founders like my friend Matt, who want to save the time they normally have to spend before/after a production deploy to avoid hearing embarrassing support issues day later. 🙇
- Sites that change infrequently. Teams which manually deploy their code (whether via a git push heroku master, cap production deploy, or `ssh my-production-server` and `rsync`). 💪
- Websites where some breakage is OK 💁, and not catastrophic. Marketing sites, blogs—sites without a “Buy Now” button and shopping cart flow. A bug on production might distract one person for an hour or two, but won’t block content creators or folks depending on your service to complete their daily tasks
Caveats aside, these services will get you pretty far, and they are fast becoming suitable for growing teams, and some work smoothly with continuous integration, git pull request flows, and let you implement advanced site navigation and on-page JS injection.
If you want to make a bet on the future improvements in customizability of these services, maintain less code, and get some tests in place ASAP, it may be worth rolling the dice on a future re-implementation and growing with these services—even with a larger team.
Enough Bloviating Already, What are the Tools?! 😈
Clicking, seeing, verifying 👉
Two amazing tools have grown to great utility over the past couple of years—Testim and Ghost Inspector.
Testim is a Chrome-integrated test authoring tool and web service started by Oren Rubin, who knows who he’s building for extremely well—former software architect spinning up the front-end team at visual website builder Wix, director of R&D at visual testing and comparison service Applitools (which it integrates beautifully), and has experience building services at scale.
Testim’s interface absolutely blew my mind when I first saw a demo. Perhaps drawing from his front-end experience at Wix, Oren’s built a visual test editor which lets you drag and drop steps, debug tests live-running in your browser, and even compose little Javascript snippets as steps themselves. The recording even uses multiple locators (explanation) to improve its selection over time.
Testim has an integration with Sauce Labs, a service which provides cloud VMs with browsers like Chrome, IE, Firefox, iOS and Android, and Applitools, the visual testing service with a seriously powerful comparison interface.
I’ve found Testim useful when working on frontend features that require a few manual steps to get to the meat of whatever you’re developing on—I’ll just record myself performing the steps and run them with F8 after each change. My dream is to hook that up to auto-run when LiveReload activates— making seeing the results of a change just a matter of saving the code (similar to what you might notice Notch does when livestreaming his game development — he’ll start a session out by writing a hook that jumps to the feature he’s actively working on).
Ghost Inspector, like Testim, has a Chrome-integrated test recorder. While the editing interface once you’ve finished your recording is less user-friendly than Testim’s, Ghost Inspector’s overall on-boarding flow and documentation is quite nice. 👻
Ghost Inspector supports Chrome and Firefox, but doesn’t seem to integrate with other browser VM providers like Sauce Labs like Testim does.
Both Testim and Ghost Inspector can be set to run as part of your CI system, and have fairly comprehensive APIs. Aside from testing their recording and editing interfaces, I haven’t used either in a production testing pipeline, so please reach out if you have experiences with either on a real project / team!
Visual Testing 👀 (or, “Dear Robot: Please Just Look at Everything”)
In terms of straight up visual testing — “please just look at my webpage and tell me if I broke it”—there are now a couple of solid options.
Applitools Eyes 👀, a very powerful tool which I’ll cover more advanced usage of in a future post, also has a super easy-to-use Chrome extension that requires no coding to use: Applitools Eyes Express. In a single click you get a visual test for the current webpage and can compare it against baselines from the last time you clicked. My brother and I tried building a similar tool for a hackathon and found sub-pixel rendering differences made false positives a real deal-breaker—Applitools solves that using some interesting machine learning / computer vision under the hood, and even uses that to aggregate changes across multiple tests/pages.
Applitools Eyes’ screenshot comparison interface is the best I’ve seen by a long-shot–truly the Photoshop of visual test management—the deeper you get into using it the more powerful you’ll find it. You can click the “Radar” button to point out where changes are, draw a region behind which any differences will be ignored, and navigate through steps, accepting and rejecting changes as you go along. Eyes has comprehensive keyboard shortcuts, making it easier to go through multiple tests and approve/reject changes. Again—more on Applitools’ main offering & Selenium integration in a future post.
Percy.io, edging slightly towards the “code required” side of the spectrum, is a fantastic option for static site generators which output static files and certain front-end frameworks. Percy is a visual testing service which provides a slick approval workflow integration right on GitHub—you can turn a red test failure from your build containing a visual diff into a pretty green dot with no re-run required. If your site fits the bill for its model of running, and you don’t need x-browser coverage, Percy looks like a GitHub Flow-using team’s dream come true.
Know What—Screw it, Let’s Pay Humans to Test My Website
RainforestQA is unlike other automated and manual QA services out there, in that they have actual humans test your website. Writing testing steps is much simpler than writing Selenium steps—because a real live person will be parsing them and navigating their way through your website.
If your website has critical paths (login, “Buy” button), Rainforest is a great tool to make sure they don’t break before or after a deploy. When first trying it out, I was surprised how quick it was to go from their homepage to reading feedback from a live person who used my website.
While they use manual testers, the team behind Rainforest understands the development mindset, with an extensive API for triggering builds and using their results. With a healthy recent round of funding and a growing customer base, it’s worth watching Rainforest for ongoing developments in making your site not break. (Founder Russell Smith is also a fun follow on Twitter!)
Go Forth and Don’t Break Stuff
It’s never been a better time for writing quick, automated tests. Coming from starting to look at automated testing two years ago—writing code and waiting a minute to watch the result, debugging broken test step code after a code change, and struggling to find ways to lock down bugs with subtle visual changes, I’ve been consistently blown away by the tooling which is now becoming available.
I’m hoping to keep this post as an up-to-date source for what’s new in the little-to-no-code automated testing world. Reach out to me @bcjordan with what I’ve missed, cool new features of the tools mentioned and any personal experiences and I’ll try to keep this fresh.
Curious what else is out there? On a larger team, with critical products? Either follow here on Medium (just click the green follow button @ the bottom of your screen) & @bcjordan and I’ll give you a heads up when my next post is ready.
Let’s learn how to save time and have sweet robots test our code together! ⚡️
