Engineering for Experimentation

Image for post
Image for post

At Pioneer Square Labs, our culture of experimentation must extend to our spinouts. When I talk to the CTOs of our spinouts, I am surprised by how much their teams need to keep experimenting. Even though we’ve launched a company with a great business hypotheses, it’s still just a hypothesis. We’ve only succeeded if we’ve instilled a culture of obsessing over building something people want (and nothing else). That requires experimentation — without over-engineering.

TraceMe just launched Tally — v2 of their take on super fans engaging with the sports they love. After launching a product targeting small businesses, Lumatax (another PSL spinout) immediately turned to a new customer — accountants. Boundless will soon serve more customer segments than just newlywed immigrants. Lastly, NextStep is developing v2 of their app while supporting a trial run on v1 with a live customer.

Frequent customer and product changes require designers and engineers to adapt quickly while keeping quality high.

Here are some challenges we’ve encountered, and ideas for staying ultra-agile while experimenting.

Write as little code as possible. Serdar, the CTO of Boundless, explains that tools like Airtable and Trello that interop well with each other are great for keeping your internal users productive without writing a line of code. Often, you can have humans do the work while you optimize software workflows. We, as developers, need to think like marketers and designers to iteratively test features we are considering using conversations and paper wireframes before we build them.

Be clear why you’re writing code. Clarify with the team what phase of the project you are building for: POC, demo, or MVP. Proof of concepts and demos can be disposable code. MVPs need to be stable — and take much longer. Write mini-specs for clear communication.

Many projects involve data science work. Hopefully, that can be decoupled and refined before turning it into production code.

Use new platforms. Try to pick the right tool (programming language, architecture, or cloud platform) for the job, rather than the one you know best. It’s a balancing act. It may be slower at first, but applying new technology is exactly why we’re innovating in the first place. You don’t want an esoteric tool that is going to go obsolete, or be hard to hire for. You also don’t want to end up with a bunch of different platforms (we’re guilty of mixing everything from Google Cloud, AWS, Heroku, to Azure). However, remember we’re optimizing for experimentation. IMHO, a successful startup engineer is equal parts expert and learner.

Dave Peck says “PSL core engineers need to be opinionated polyglots. But there’s a balancing act: if we always use our go-to tools, we’ll hit targets sooner but miss opportunities down the road. On the other hand, if we’re always learning new tools as we tackle new targets, we’ll probably go too slow. It’s just another trade-off to juggle.”

Design modular architectures. It’s hard to deal with changing APIs and iterative development. It’s ultra-important to design modular code, so you can replace parts of the airplane in flight later (if it’s not scrapped). Of course, this is good practice anyway.

Rob McDaniel adds: “If you don’t spend enough time on good design patterns, you will absolutely destroy any chance of experimentation because your system will be so brittle that it will break anytime you touch anything. On the other hand, if you over-optimize too early you’ll make design decisions that you end up being wrong about, and you’ll never ship. I think that the magic is the balancing point between the two. For me, my mantra is ‘avoid decisions.’ If I can make a convincing argument to put off a decision, then it’s too soon to make it. Every decision we make ‘locks us in’ to a new path, and getting locked into a path is the antithesis of experimentation. A clean and flexible design is the key to fast iteration. The trick is that this requires an EXACT amount of forethought, designing for ambiguity and change by making smart use of good design patterns, and knowing that done is better than perfect — but only most of the time.”

I think “avoiding decisions” is a great way to put it. If it’s not clear what you need to build, wait until it is clear. Sedar has another way of putting it: “If you have a choice, make dumb systems. Whenever we built systems that try to be intelligent about a complex process, we ended up fixing them a lot.”

“Choose powerful over scalable,” is how Ryan Kosai put it.

Ship first. Change is inevitable. We do short one-week or even daily sprints. Be careful not to burn yourself out on any one spike. It’s hard to weigh spending time on things like continuous integration vs, what I call “coding in front of a real studio audience.” First, experiment in front of users, then shore things up for MVP. “Speed is life!” says EIR Glenn Wisegarver.

Image for post
Image for post

Delete code often. It’s a pain maintaining barely used code. Scrap what you can as soon as you can. Delete it and commit, don’t comment it out.

Hire for drive, curiosity, and diversity. Farhan, VP of Engineering at Lumatax says that drive & passion outweigh experience. Similarly, Boundless sees team diversity as a means to preventing blindness to customer problems. I hear this as a challenge to those of us that have been around awhile — don’t let “expertise” be blinders.

Don’t forget monitoring. Another tip from Serdar Sutay— “You’re not done after shipping your code to production: we’ve introduced a specific stage to our development workflow (Monitoring) to reflect this.”

Image for post
Image for post

One thing I’ve been thinking a lot about is how experimentation is a disorganized creative process, not a pipeline. What you might think of as “engineering quality” has to take a back seat to figuring out the right thing to build. Traditionally, this is more of a design and marketing exercise. For startup developers, it requires an open mind, constant learning, and even cutting corners — but with a knowledge of the trade offs.

Image for post
Image for post

Principal Software Engineer at @PSL. Bike racer with TacoTimeNW.bike.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store