Cross-browser Testing Through the Ages

Browser-testing company and Mozilla partner Sauce Labs grew out of the automated testing framework project Selenium in 2008. Since then, they’ve run more than a billion tests against countless browser/device combinations. All developers have struggled with cross-browser compatibility and tools like Selenium and Sauce Labs streamline the process. They also help keep the web open and accessible — the more people who can use your site, the better.

We sat down with Sauce Labs founder Steven Hazel to talk about cross-browser testing through the ages (in web timescales), Sauce Labs, Selenium, internet health, machine learning, and more.

How has cross-browser testing changed over the years?

When we got started I think most browser testing was either manual or record and playback — just trying to make manual testing more efficient by automating the steps. But it was still essentially a manual process. So for example, if the application structure changed, you might have to go re-record all your tests, which was a major pain. Or if 20% of a test suite failed because of something like that, people would think, hey, we just saved 80% of the effort, and they’d make up the difference with manual testing. Since then, testing has gotten a lot more fully automated. It’s become part of continuous integration, and continuous deployment pipelines. You script what ought to happen. And you script in a much more sophisticated way.

Why do we often overlook mobile browser testing?

The fundamental thing is if you’re a developer, you’re sitting at a desktop when you’re coding. It’s convenient and easy to fire up a browser and take a look at what you just did. And it’s less convenient to take your phone out of your pocket and figure out how to get it pointed at your dev site. The answer to that is automated testing, so all the platforms you don’t have handy, including mobile platforms, get checked thoroughly with every change. Without automated testing, the unfortunate but understandable tendency is for mobile to be an afterthought for web developers.

What are the most overlooked platforms?

I would say the most overlooked browser is Safari. It’s a small slice of market share compared to the other browsers and I think people feel like they can get away with not testing in Safari because it’s a WebKit browser. Whereas nobody thinks because you’re testing in Chrome that it’s okay to not test in Firefox. You gotta test in Firefox. But people think they can get away with not testing in Safari, not realizing that is significantly different from Chrome in some important ways. Because of that, I would say most often if you’re going to see a layout bug that only affects one browser, it’s Safari.

What makes an effective test suite?

First, reliability. When your test suite is reliable, and you know that a problem with your test suite is likely to mean that there’s a bug in the app, you come to rely on it in a different way. You make it part of your deployment pipeline, you have those tests gating your path to production, and you develop a real confidence through operating in that way. You can deploy small changes frequently and know that you’re not going to break something. If you don’t focus on the reliability of your test suite, it gives you dramatically less confidence in your app, and you wind up continuing to depend on manual testing. Your pipeline slows down, and now you’re not deploying small changes frequently, you’re deploying bigger changes more infrequently, and that problem snowballs and you’re back in a waterfall mode of development.

The second thing is speed. When you first start testing, you build a little test suite that runs fast, and everything’s great. As you add more features, or you add more tests, you build up a bigger and bigger test suite. The wonderful experience you had early on with a small test suite that ran fast and gave you quick feedback goes away. Your tests take hours and hours. So coming up with reliable ways to run your tests in parallel so that you can get through the test suite rapidly gives you a way to hold on to that fast feedback experience.

How do you build tests to run in parallel?

With something like Sauce Labs, you can run hundreds of tests at the same time instead of cranking through your suite one test at a time. It’s important that you build your test suite the right way so that two tests running simultaneously aren’t going to affect each other, or they’re not dependent on one test creating some object in your system, or setting up some account that is going to be used by a later test. The tests need to be self-sufficient.

What are some common things people overlook when testing?

There’s a gap between testing the functionality and the visual appearance. At a lot of places visual testing is still manual, or people aren’t doing all that much of it. That’s an interesting place where there are tools to recommend. We partner with a company called Applitools that makes good visual testing tools that can integrate with your Selenium tests. They take screenshots and can help to automatically pull out significant visual differences that somebody should take a look at. Automated visual testing is getting better and better, but there’s still a human element and it pays to take the time to really check the visuals on different browsers.

Actually testing on real devices is important, too. If you have the space and time, it helps to invest in a physical test lab full of new and old devices you can actually test your app on. It’s not always easy to manage, but it becomes necessary at a certain point. We recently acquired a company called TestObject that does mobile testing on real devices in a scaled, automated way. I’m pretty excited about that because it could eliminate the need to build out a physical test lab for a lot of people.

What advice do you have for new web developers about testing?

Think of the most important feature that your app has. Take a day and figure out how to do automated testing to test that one feature. Then just start running that every time you make a change. The sensation you get with that quick check mark is satisfying. And the one time you do break it, and it catches it for you, you’ll get the value of testing. Just start from one good test that tests something important, and that works well and reliably, and it fast, and then add tests from there sort of one at a time, incrementally as you build your features.

What are you looking forward to in testing and software development?

We’re just at the beginning of some of these new approaches to building software — continuous delivery, service virtualization, container-based microservices. They’ve changed the whole concept of how you build software at scale, how you deliver it, how you test it. A lot of those things, I feel like we’ve just figured out the first good idea. How far we can go from there? One common thread is, we’ve gotten a lot better at encapsulating a piece of software and being able to deal with it in isolation from everything else. I think there’s a lot more we can mine in that vein.

The degree to which dev process has improved over the past eight or nine years is super exciting. And I think that we’re still in a transitional phase here. Not everyone has completed their transition into DevOps, into doing continuous delivery. And I think that as more and more development organizations make that move, it’s going to be exciting to see just how much more capable they become and how much more sophisticated software gets.

Why is cross-browser testing important for maintaining an open Internet?

Cross-browser testing is the only way developers can be sure their web apps aren’t dependent on the quirks of any particular browser implementation. When web apps are built on a common set of widely-supported standards, they can reach more users on more devices, the developers of innovative new devices can more easily bring the web to their users, and end users benefit from a competitive ecosystem that offers them choices and new possibilities.