Percy, 3 years and 75M visual diffs later

Three years ago, I got up at 6 am on a Friday morning, excited to press “go” and introduce Percy to the world. 🚀

I think I expected to get some coverage and initial interest—and I made sure the infrastructure could handle a big traffic spike, just in case—but what ended up happening far exceeded my expectations.

Over the course of the day, I watched our “Show HN” post climb quickly to the top of Hacker News, and it stayed at #1 the whole day. 😲

We continued to be in the Top 10 even days later, and ended up bringing in over 35,000+ unique visitors from around the world. Developers everywhere seemed really curious to learn what “continuous visual integration” meant!

In that first post, I penned a lofty mission for Percy: make continuous visual integration so simple and fast that it becomes a mainstream engineering practice.

Since then, a lot has changed. Though we have a much deeper understanding of the problem and often use different words today (and we now have a real marketing person, instead of me!), the mission remains the same: enable teams to ship products faster with confidence in every visual change.

Although our space is still evolving, and we all still struggle a bit explaining what “visual testing” is to those outside the field, it’s growing and growing quickly.

How do we know that?

Since Percy’s launch in May 2015, we’ve rendered over 75,000,000 screenshots and visual diffs.

75M not only represents the number of times we’ve opened a browser and rendered screenshots (over 50 years in raw compute power), but we measure this in terms of confidence gained in each release, and countless hours saved doing manual visual QA. It shows a glimpse into how we’re starting to change developer and design team processes and help people gain new levels of confidence that would be impossible to reach otherwise.

In getting to this milestone, my co-founder Tim and I have committed to thoughtfully building Percy from the ground up, keeping our mission top of mind, and building out a crafty and diverse team who embodies it. Each year we’ve gotten a bit closer to that goal, and are learning more about the intricacies of the challenge and opportunity at hand. We still have a long way to go, and it’s getting more exciting each day!

Looking back, I wanted to share a few of the things that we have learned in building the product and business over the last three years.

2015

In our first year of business, we spent a lot of time listening to our early customers who were passionate about what we were doing. We built essential integrations to help get visual testing into day-to-day engineering workflows, and learned how teams of all shapes and sizes experienced the same pain.

• Once we’d successfully built initial support for Rails + Capybara tests, static sites, and the top CI services, we were ready to launch in beta (or so we thought).
• After re-architecting Percy to overcome some early bugs and scalability issues (for example, we found a GitHub user who had forked 16,000 repositories!), we sent out our first 50 beta invites. Before long, we had our first early customers, and they started seeing the value of automated visual testing.

• Towards the end of 2015, I was excited when Percy hit the 50,000 visual diffs rendered milestone (also wow, how we’ve grown). We also announced support for parallelized test suites, a core Percy feature that we just take for granted now, and officially launched Percy for all!

As you can imagine, we learned a lot in those first months. 🤯 We learned how to start talking about Percy. We started to see if we could build a business that could survive. We learned how to help developers get visual testing in the door on their teams. And probably most importantly, we learned that people were as excited about solving this problem and improving their workflows as we were. I’m not going to say it was all easy (spoiler: it wasn’t), but it was definitely exciting.

2016

In 2016 we introduced more integrations, launched some milestone product features, made significant snapshot rendering improvements, and open sourced the Percy front-end.

• The most significant milestone for Percy post-launch was the implementation of the #1 most requested feature at that time–visual regression tests for responsive designs. 📱 This is still a huge part of Percy’s value today—knowing immediately when a change affects your mobile site, without even having to view your site on mobile.
• We also launched support for a core feature of Percy, animation freezing, which is a critical part of a stable visual testing workflow. We also rebuilt and refined our custom rendering infrastructure many times to improve snapshot rendering at scale.
• We also hit the 1M diffs milestone, and ended up more than quadrupling that by the end of the year.

• In the middle of 2016, Sentry sponsored development of the python-percy-client, and we got Percy integrated into Python and Sentry’s rapidly growing frontend applications. Thanks, Sentry!
• We also launched support for Ember.js and Buildkite in 2016, two integrations that are near and dear to our hearts as we use both within Percy’s stack. I’ve personally really enjoyed being involved in the very design-centric Ember community (see my recent talk, Pixel by Pixel: Visual Regression Testing in Ember), and we had a great time hanging out at our first booth at EmberConf recently as well:

• As big open source advocates ourselves, we decided to open source Percy’s entire front-end web application at the end of 2016.

In 2016 we dove further into the nitty gritty of Percy’s rendering infrastructure and snapshot stabilization. We validated a lot of our model for how Percy’s rendering infrastructure would work and scale to support truly continuous visual testing, and how to support complex parts of web applications that can make visual testing hard (like animations, canvas elements, dynamic dates and data, etc.) We pour a lot of continued research and development effort to making sure we can robustly render screenshots across modern browsers and devices, and highlight the most relevant visual changes.

2017

Last year was the year of expansion and exciting early engineering team growth, getting as much feedback as we could, upgrading our look and dramatically refreshing and simplifying the Percy UI/UX for our growing customer base.

• Our mascot, Percy the Perceptual Porcupine, got a new look! And of course, in implementing redesigning our own website, we used our own product to ensure all visual changes were intended.

• Our Storybook integration was a huge 2017 initiative, bringing visual testing to React and component libraries.
• In May, we launched as one of the first few launch partners on the new GitHub Marketplace.

• With the core Percy functionality becoming more robust, we spent much of 2017 redesigning Percy’s visual review UI to make each visual review flow more efficient, and adding new integrations and browsers to our stack.

After the first two years of technical and business challenges, 2017 was the year I started to feel Percy had serious traction on the business side. With this momentum, we met many great investors who shared our vision for where DevOps and visual testing can go, and we successfully raised our first and second rounds of funding. Our amazing early team grew quickly, shipping multiple redesigns and new features, and by the end of the year, Percy felt like a completely new product and company.

2018 (so far)

In the past few months we’ve doubled the size of our team, built new functionality that exponentially increases Percy’s value to teams, and we’re just getting started!

• Just a few weeks ago we wrote about our new single-snapshot approval processes which allows teams to iteratively approve snapshots as a PR develops. This update and others to our UI were key to advancing the experience of day-to-day visual review workflows.

• As we continue streamlining the visual review process, we’ve introduced several new core features and more subtle updates. We’re always trying to make visual reviews easier, so we added a simple-but-snazzy diff toggle shortcut, check it out!
• We’ve started betas of GitHub Enterprise support and our cross-browser testing solution, and have some exciting new integrations coming out soon.

The evolution of Percy wouldn’t be possible without the support of our customers and lovely community. Watching developers experience the “AH-HA!” moments of automated visual testing is what keeps us going. 💜💜💜

I think Trek wrote our “unofficial motto” with this one:

And my personal favorite from earlier this year (thanks for the kind words, Vaidehi!)

😊

I couldn’t be more thankful for the early customers who saw the value in this new visual testing paradigm. They dealt with our early bugs, gave invaluable feedback, and saw the early promise of what Percy could do. We continue to dig in deep to support and learn from our customers, and I’m humbled that many of those who joined us in the first few days and months have stuck with us ever since.

What’s next?

It’s hard to convey how excited I am for the next phase of Percy. I think we’re going to look back on these days as engineers and designers, and wonder how we ever got by without truly automated and continuous visual testing. The key for us going forward is, it has to be ubiquitous and smart too, and that’s what we aim to do.

We’re taking the past three years’ learnings and moving forward to make visual testing a more accessible and invaluable process for product teams. We have lots of major updates in the works that we’re excited to share with you in the coming weeks and months, which we hope will impact the way companies of all sizes design, develop, and deploy software. You can follow @percy_io on Twitter or subscribe to this blog to keep up with what we’re working on.

As always, thank you for the support you’ve given us over the last three years, and here’s to many more! 🎉

Percy is a visual testing platform that gives your team confidence in every visual change before it’s shipped. Learn more at https://percy.io.