The Evolution of Testing User Experience
I’ve been involved in developing web and mobile products for over a decade now, and one of the most consistently tricky problems to solve has been user testing. The exact words we use to describe this process have changed (i.e. user experience testing, quality assurance testing), but the end goal is always the same: to ensure that the product works as intended.
The Old Way
The technologies we use to build products has become more advanced, and so have the techniques and services we use to test those products. One of the most rudimentary techniques is to have a dedicated testing engineer write out a testing plan for each step function of the product. These are (fundamentally) laborious to create, maintain, and execute.
Whenever a product was scheduled to be released, a team of humans would go through and complete each test, ensuring that the actual results lined up with the expected results.
Introducing Automation
Over time testing has become more commonplace and a whole range of services has proliferated around it. For example, UserTesting.com is one such service which allows subscribers to obtain videos of real people attempting to navigate a product by following a prescribed testing plan (similar to the one pictured above).
Software development has also become more complex, impacting user testing in particular with the influx of Test Driven Development (TDD) tools. TDD is based on the premise that you should write tests for software before you write the software itself, and the tests can run automatically whenever you want. The tools around this have become so complex that they can even test if the correct things are displayed on screen after a button is pressed.
No testing method is without downsides, and TDD is no exception. To test that pages render correctly, tests must become more and more specific, which greatly increases the effort in maintaining the tests. The overall development environment is incredibly complex as well.
What’s Next
There is a slew of next generation user experience/testing services available now. One such service is FullStory, which silently records your users sessions, and then allows you “play back” any given session. The benefits of using this service can be felt beyond the product team; the information can make an impact on design, support, and sales teams as well. Cross functional teams within any organization can draw inspiration to improve at large, after being able witness ‘replays’ of customer experiences. Sales will want to know if a hot prospect is struggling to get a core feature working. Support is trying to help a customer based on limited information, but now they can replay the session and get on the same page as the customer. Design teams can see how a particular menu or interface is being used by real customers.
The end goal of these techniques and services is delivering a better Customer Experience:
“… the product of an interaction between an organization and a customer over the duration of their relationship. This interaction includes a customer’s attraction, awareness, discovery, cultivation, advocacy and purchase and use of a service.[1] It is measured by the individual’s experience during all points of contact against the individual’s expectations.”
With this understanding, it is crazy that so little emphasis is placed on the billions transactional emails sent to customers each year. I specifically call out transactional emails, (no, not the newsletters you send with Mailchimp), because by their very nature transactional emails are triggered based on user-product interactions. They are a core part of the Customer Experience, which companies pour money and resources into with complicated development systems and testing software.
The main reason transactional email has been ignored is because it’s so damn hard to test. Services such as Litmus help answer questions like “what will my email look like on an iPhone 5 running the GMail App”, and “does this image show up in Outlook 2003 on Windows XP?” However this kind of service has primarily been accessible to marketers sending newsletters. An entirely different rant, but it seems logical that the people within an organization that care about customer experience (primarily marketing and sales) should have the most ownership. That’s not the case with transactional email. Typically, transactional email templates are hard-coded HTML files in source code, making it incredibly hard to test whether they contain the right information, let alone if they render correctly on your friend’s iPhone.
A New Competitive Edge
This is where sendwithus steps in; closing the loop on user testing in transactional email. At it’s core, sendwithus is a template management system for transactional email testing — like WordPress (collaboration, versioning, analytics), but for really complex email templates. One of the more alluring core features of sendwithus is the ability to store an exact copy of each email that is sent out. If your support team wanted to know for example, what your friend’s receipt looked like on their iPhone, with all the dynamic data like purchase time and order values included, sendwithus could show it. Combining Litmus’ and sendwithus’ capabilities, you could pull up a complete rendering of any transactional email (and all its content) to see exactly how it appeared on any combination of device and email client.
Remember how product, design, marketing, and sales all received a ton of value from seeing exactly what the customer did on a website via FullStory? Now we can do that for the billions transactional emails sent each year. That’s the evolution of user testing; seeing things exactly as your customer has seen them.
Excited to start testing your email experience? Learn more about how we do device testing with “Your New Email Super Power” or check out our Team Collaboration Solutions.
