Maker Hunt’s AMA with Kyle Rush Director of Frontend Engineering and Optimization at Hillary for America
Kyle is a technologist, speaker and consultant living in New York City. Currently, he is the Director of Frontend Engineering and Optimization at Hillary for America. Kyle’s work has been featured in the book “A/B Testing: The Most Powerful Way to Turn Clicks Into Customers” available on Amazon. Before joining Hillary for America, Kyle was the Head of Optimization at Optimizely. Prior to working at Optimizely, he was the Director of Technology at The New Yorker where he helped re-platform and re-design www.newyorker.com and design a metered paywall. Previously he was the Deputy Director of Frontend Web Development at Obama for America.
Hi everyone! Super excited to chat with you all today. I’m more than happy to take questions about a/b testing, engineering, user experience, etc.
Q. Can you give an example of one of your most successful test and what important insight you discovered? — Ben Tossell
I wrote this up in a growth hackers AMA recently. take a look here:
AMA, Must Read] I'm Kyle Rush (@kylerush), Head of Optimization at Optimizely. Ask me anything! As the Head of…growthhackers.com
tl;dr: the most successful test I would say was not necessarily a winner, but one that helped me understand the importance of sampling.
Q. What’s your feeling about Multi Armed Bandit as a statistical approach vs traditional A/B Testing? — Michael Buckbee
IMO mutli-armed bandits are just a way to mitigate risk, but the problem is that they will likely require a larger sample size than is otherwise necessary. i rarely use multi-armed bandits. pretty much the only scenario i would use one is if i had to deploy an experiment late in the day that was risky and would reach a large sample size by the time i was sleeping at night. that would allow the bandits to switch to the winner while i was sleeping. otherwise, i never use multi-armed bandits or auto allocation.
Q. Are all of your decisions data-driven? For instance, are there times where you make a decision that somewhat contradicts your data because it “feels” like the better decision to make? If yes, can you tell us more about the intuitive side of things with optimization. — Eric Willis
Yes, definitely. From what i experienced in publishing, it’s a world where everyone copies each other. If the times does, everyone else will. It’s a very competitor centric thought process in terms of roadmap. Publishing companies are generally speaking very hesitant to try anything new unless a competitor has done it to success.
Q. What is the most important optimization an e-commerce website should take care? — Savvas Zortikis
That’s a really hard question to answer without context. i think for companies that are just starting, my best advice would be to focus on pages/screens that are the last step in the funnel. these pages have a higher conversion rate, thus require a smaller difference in the conversion rate and will get you winners quicker.
Q. Do you think more companies will take the metered paywall approach? — Eric Willis
The paywall at the new yorker is doing well, btw.
Q. What lessons are you taking from the Obama campaign that you’re now applying to Hillary’s campaign? (the important ones) —Eric Willis
I’m not supposed to talk much about our work in the middle of a presidential race unfortunately. What i will say is that the Obama campaign taught me how important it is to sculpt a team that can move at lightning speed. This is helpful in responding to the news cycle, changing poll numbers, etc.
Q. What were some of the most important lessons (in terms of optimisation) that you learned running the Obama for America campaign? — Ben Tossell
The truth is I learned more than I can even remember! I feel like personal growth on campaigns cannot be matched elsewhere. I think understanding statistics is pretty important because this might help you understand erratic or unexplainable results in a/b tests. Aside from that I’d say always be testing!
Q. How can we at Product Hunt help ? We want to help spur interest in gov’t/politics among people in technology — Erik Torenberg
Great question. I would say interesting examples of gov’t api’s would be a great way to get people interested in how policy affects them. Interesting examples of gov’t api usage*. As in, build a cool interface using gov’t api’s that gives people a new way to look at data.
Q. What is the minimum traffic a website should have in order to start the optimization process? And what about the minimum amount of email subscribers? I know that you made a lot of email A/B testing in the Obama campaign. — Savvas Zortikis
I would encourage you think about this less from a sample size perspective and more from a “how much difference can i detect” perspective. The reason is because sample size is just a simple formula. There really is no minimum number of visits requires. It’s just a matter of how much traffic you have and how much of a difference in the conversion rate you can detect. here is a sample size calculator to help you:
Visual, interactive sample size calculator ideal for A/B tests.www.evanmiller.org
And here is an article i wrote about this:
As the head of Optimization at Optimizely I often hear in the startup community that small startups cannot run…kylerush.net
Q. What would you be doing if you weren’t in the optimisation game? — Ben Tossell
I’d probably be working at a dog rescue. I really, really, really love dogs.
Q. How do you decide which tests to run first? — Ben Tossell
I think this is the second hardest problem in optimization. There really is no science to it. This is where we need to use our critical thinking skills. Generally speaking it sounds to reason that big changes have a better chance of impacting the conversion rate (for better or worse) so i generally like to start there. That being said, you should weigh the ROI based on the time to implement a variation and the estimate impact.
Another way I like to approach this is creating a first test that completely strips the interface/page of any non-critical UI component. That usually gives a big lift in the conversion rate. Then I can one-by-one add the old UI components back in with an experiment to see who much impact each one had. This will help you understand what should and should not be on a page.
Q. Do you have a set of top resources you would recommend for optimisation? — Ben Tossell
I wish I had a better answer about resources. I get asked this a lot. I try to read growth hackers regularly. I follow smart people on twitter. I read hacker news. Other than that I don’t have any great suggestions.
@lukew is a great person to follow on twitter
Q. What question doesn’t come up in AMAs that you’d like? — Ben Tossell
This would probably be something like “what’s the number one thing I should focus most of my attention on?”
A lot of people tend to get sidetracked with buzzwords and such which is very distracting.
So the number one thing you should focus your attention on, without a doubt, is being more creative. Analytics is something that can be learned with time. Process is something that can be learned with time. Statistics is something that can be learned with time. Creativity is extremely hard to teach and it doesn’t matter how analytical you are, your testing program will not be a success without a lot of creativity. Really try to put yourself out there and be inspired by things other people are doing and learn to think outside of the box.
Thanks so much for doing the AMA with us
The pleasure was mine! Feel free to hit me up on twitter if you want to chat more.
Next AMA is with:
Mat Carpenter— Founder of Ship Your Enemies Glitter and other projects — 3pm EST 16th June
If you liked this AMA please Recommend ☺. Keep an eye out on our Twitter — @MakerHunt for details/updates.
Special thanks to Kyle and everyone who participated. If you’re a maker with a product on ProductHunt, be sure to sign up to participate in the AMAs and connect with others.