Scaling secure tunnels for browser testing

Shane Chapman
Dec 14, 2018 · 4 min read

Testing is critical to Walmart’s successful e-commerce business. There are 100s of development teams across the world writing code and all of this code needs to be tested before it sees production. But trying to support every browser and device version to test on is a herculean task in itself.

We achieve this via a vendor, Sauce Labs, which does just that. You give them a selenium script with a browser & version and they’ll execute the tests on that specified browser. Most of the time, these tests make service calls back into our internal network which causes a new problem. We can’t open up all of our test APIs to the public.

Sauce Labs has Sauce Connect™ tunnels which allows us to create a secure connection into our own network!

Sauce Connect™ is a proxy server that opens a secure connection between a Sauce Labs virtual machine running your browser tests, and an application or website you want to test that’s on your local machine or behind a corporate firewall

We have setup multiple DMZ boxes to handle this inbound traffic from the Sauce Connect™ tunnels.

While this sounds like the perfect setup, there is still some plumbing that needs to be done to ensure scalability and availability. We also ran into the following 2 issues:

  1. Sauce Connect™ tunnel would become unresponsive with no notification
  2. Suace Labs recommends to restart each Sauce Connect™ tunnel once every 24 hours

Introducing Lookout & Raven!

  • Raven is an application that launches a Sauce Connect™ tunnel in the DMZ
  • Lookout is an application that monitors the health of the Raven(s)
Lookout — Raven — Orchestrator Architecture

A Raven’s sole purpose is to start and stop a Sauce Connect™ tunnel. The health check logic is pushed off to the lookout application because Sauce Labs rate limits authenticated requests.

For authenticated requests, these limits are imposed for individual user account

The limit is 3. If we have more than 3 Raven’s with the same credentials for a tunnel, we would rate limit ourself checking the status of the tunnels.

Instead, we push this logic onto the Lookout application. A single Lookout application will be deployed for a cluster of Ravens that share the same tunnel id. This allows us to check the status for a collection of tunnel ids with the same credentials. Once Lookout acquires the list of active tunnels, it writes these tunnels to file. At this point, we can have all of our Raven’s check the file to see if they are still active tunnels. If they are NOT in the active list, they will terminate themselves.

All of our Ravens and Lookout applications are controlled by a single configuration file and process management which we call our Orchestrator.

Orchestrator solves the following problems:

  • availability requirements by having the process manager restart the Raven any time it terminates itself
  • requirements by deploying to multiple DMZ machines
  • scalability requirements by allowing us to add/remove/restart tunnels without impacting other running tunnels

Impact of these applications

  • Zero manual restarts for unresponsive tunnels since the Lookout and Raven implementation (Previously needed a manual restart ~4 times a week)
  • No customer test case issues reported because of tunnel outage (Previously received ~1 customer issue a week)
  • Zero impact on existing tunnels or running test cases when starting new tunnels

Conclusion

Ensuring that code is tested on multiple browsers and versions is very important to any company with a web presence. There are so many different versions of browsers that setting up the correct infrastructure to handle that is a great undertaking. Take advantage of the tools that are already available and use them to your benefit.

WalmartLabs

Using technology, data and design to change the way the…

Shane Chapman

Written by

Staff Engineer WalmartLabs — Take things apart that aren’t broke and fix them

WalmartLabs

Using technology, data and design to change the way the world shops. Learn more about us - http://walmartlabs.com/

Shane Chapman

Written by

Staff Engineer WalmartLabs — Take things apart that aren’t broke and fix them

WalmartLabs

Using technology, data and design to change the way the world shops. Learn more about us - http://walmartlabs.com/

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store