Postcards And The Offline A/B Test

**I’m launching a newsletter in 2022 called Personal Finance in the Blockchain Age. It would make my day if you sign up.**

Things are heating up. The week moved by in a blur. There was a lot that happened, but my main focus has been working on ways to attract customers to Visionary Rentals (we rent virtual reality rigs and offer VR events). There are a lot of digital options which we will be building out further in the coming weeks. However, I needed something physical to hand out to people when we perform demos and talk with potential customers. It was clear that we’d benefit from a postcard.

Postcards are a great way to promote a product, business, or service. Postcards and their brethren (the flyer) have been a marketer’s tool for centuries. Unfortunately, in today’s world of digital customer acquisition channels that deliver massive amounts of data, the offline tools are sometimes overlooked for not being as efficient or nuanced as digital. However, you can run an A/B experiment with a postcard just as effectively as comparing two AdWords ads. There are other benefits as well that make postcards an effective marketing tool. They don’t require an internet connection, can be placed nearly anywhere, and are well-suited for any business that offers a product or service in real life. Plus, they are inexpensive (about $0.20 each when you buy 500), and you can give out hundreds in a day with minimal incremental cost.

Setting up A/B experiments when you’re first getting started is difficult. It takes so much effort just to build that first experience (ie the control), that you may not have the effort or creativity to create the variant at first. This was an unexpected (or perhaps forgotten) challenge.

I decided to forgo an A/B experiment, because I had a self-imposed deadline 7 days away so that my business’ momentum would continue. I was also thinking that ANYTHING would be better than nothing, and we could move on from there. After thinking through the products we’d promote on each side, I mulled over what the postcards should look like. I knew that we’d iterate quickly if the postcard failed. I sketched out what I wanted on a piece of paper and took a picture. I found a designer on Fiverr ($25 + tax) who brought the design (front and back) to life, and after a few iterations, I signed off on the comp and the vector file was sent my way.

It was key that I have a copy of the source vector file (aka Adobe Illustrator file). It’s the easiest way to make changes for future versions, and from my time working in film distribution, I had experience using most of the Adobe Suite.

I tweaked the design a tad more since the language barrier with the designer made some minor details difficult to communicate. I had been researching online printers while I was waiting for my designer to complete the comps. I chose Overnight Prints, because they seemed the most reasonably priced with the best website (their image processing of the postcard was the best I saw). My rush order of 500 postcards cost just under $100, and the delivery would be in three business days. From designer hired to printed postcards delivered, the whole thing took 7 days.

I was super excited by the creation of our first postcard, and the potential impact from a planned postcard handout on Friday. My expectations were that we’d get at least a rental from the postcards, hopefully more. On Friday morning, I passed out about 80 postcards during 90 minutes in San Francisco.

I was happy with the pace, but unfortunately, we had no rentals. Even the sign-up for our Free Virtual Reality Event contest had limited response. This was a let-down, and I spent much of Friday afternoon just staring at the 4x6 postcard, dissecting all the ways it could be improved.

I learned so much.

Upon a more critical examination, I realized the postcard was sloppy, even though I’d been fawning over it since I received it. The cardstock and gloss made it feel professional. The simple color palette, different dominant color on each side, and copy looked good. However, each side had multiple selling points. For instance, on the side promoting our Weekday Special (a $60, Mon to Thur VR rig rental), I also had our general rental pricing which was a big distraction from the $10 off coupon for the Weekday Special. Even I had difficulty reading through the content at a glance. The worst part was the lack of photos or anything that explained our product.

Within 24 hours of the initial failure, I had designed a new postcard in Adobe Illustrator (not even needing to pay someone this time) that addressed the flaws I identified while conforming to my first experiment thesis — one clear call to action.

The latest iteration has been sent to the printer and should arrive this week.

For my second effort, I intend to take a more thoughtful approach. The first time I received the postcards, slapped some half-thought-through discounts codes on them, and raced to hand them out. This go-around, I’ve documented and refined my A/B testing framework to speed up the creation and experiment process so that I can maximize my time going forward. I decided to share it in case it helps someone else avoid my mistakes or at least feel comfort that you’re not alone.

Until next time…

Elliot

P.S. If you find this helpful, please share it with friends and colleagues.

Postcard A/B Experiment Framework

  1. Determine Experiment. The first iteration doesn’t need an experiment. Sometimes creating that first comp is all you need to get started. Going forward, however, always have a reason for creating a new design and use that reason to come up with an experiment thesis.
  2. Create Materials. You will need to design the postcard, but you should also consider the discount code or URL that you place on the postcard along with how you can track and measure interest. For instance, if you simply direct everyone to your main site (ie visionaryrentals.com), then you will not have any attribution for the visit. Other than maybe a bump in traffic, you won’t be able to measure this meaningfully. If, on the other hand, you create a redirect page at visionaryrentals.com/postcard (page doesn’t exist) and forwarded that url to your main URL (ie the index page) along with a URL parameter, you will know they came to your site from the postcard. Given that you need to know your tracking methods before you go to the printer, you’ll need to sort this out during this step.
  3. Manufacturing. Postcards are a pretty simple thing to manufacture. You provide your files based on pre-determined paper sizes, select your preferences, fill out the arrival date, and pay. You mainly just have to wait, though when there is an error, do not hesitate. You should call as soon as there is ANY disruption to the process. Be kind and patient, but don’t get off the phone until you know what’s happening. When I got shirts made, everything went perfect until the shipping company re-routed my package to Canada. I only knew this because the day it was supposed to arrive (the day before the event), it didn’t. Rather than rely on the tracking info (which I learned was wrong), I called and after 30 minutes of chatting, I learned what really happened about the package being re-routed. Because I stayed on top of the vendors (t-shirt maker and shipping company), I got a big discount on my order, a voucher for my next order and a second order of the shirts for free as the tshirt vendor attempted to still meet the deadline. While it was a non-ideal outcome, the vendors compensated me for the inconvenience, so there was some value.
  4. Assemble Components. I thought long and hard about how to use discount codes. I ultimately settled on printing out discount codes and sticking them to a large-batch, economies-of-scale postcards order rather than print small-batch postcards which would be two or three times more expensive. While this is time-consuming, it’s also a mindless activity that’s perfect to do while watching TV. Plus, it gives me more granular insight into how effective certain postcard locations are. After applying the discount codes, I use rubber bands to bundle up the postcards into manageable quantities. When you want to grab a stack of 100 postcards, having 10 or 20-stacks ready to go is useful.
  5. Distribute Postcards. You could go with a service that distributes postcards. I hired what I thought was a student (turns out it’s a business probably hiring students) to post one side of the postcard as a flyer around Berkeley. I didn’t get any response to the flyer or see anyone from the city of Berkeley look at the site per Google Analytics. It could easily be due to the low conversion of the postcard itself or Google Analytics not tracking well (which means my next URL-based redirect method will be more effective), but outsourcing the distribution when you have no idea what you’re doing introduces too many variables. I’ll need to re-test outsourcing once I know the postcard works. That means you may need to invest the time to hand out postcards yourself. Go to busy spots where there are lots of people, and don’t be afraid to shout your pitch. Just don’t be pushy. People are walking and while some may reach for a postcard, others may not be as open. Be respectful, and the next time they hear your pitch, they may be open to your product. Repetition is key, especially for cold sales.
  6. Monitor Metrics. Based on your metrics, determine which postcard won the A/B experiment. Once you reach a certain conversion rate that makes the postcard campaign profitable, it may be more worthwhile to stick with the same postcard and only iterate / experiment again when you see the metrics fall. Until you reach that conversation rate though, the end of one experiment will mean the beginning of another, so rinse, repeat and start back over.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store