Moving Fast: How to Launch 13 Products in 12 Weeks

Indeed University (IU) is a 12-week program for new and recent college grads joining our product development teams. The program’s goal is to instill data-driven, rapid-iteration, impact-focused mindsets to the new employees. I was fortunate to support Indeed University as a mentor and manager in 2018 in Seattle.

We believe that the best classroom is the real world. We quickly ramp up the new employees, then help them brainstorm ideas they feel passionate about solving. The participants form teams around problems that job seekers and employers face, then attempt to build products to solve those problems.

We had 13 teams form during the 2018 Seattle IU cohort. Teams in every cohort address a variety of issues, from improving job search to automating reference checks to improving job descriptions. Senior leadership staff members meet weekly with teams, challenging them to use data to show their products’ value for users.

Leads act as facilitators and mentors, not traditional team leads. As leads, we need to help teams move fast during Indeed University to take advantage of the 12 weeks they have to work on these products.

Lesson #1: What does it mean to move fast?

Moving fast isn’t about mindlessly zipping through tasks or striving for high velocity or completing a large number of code commits. It’s true that these measures can lead to quicker delivery, but they miss what’s important. We want every code commit to make a difference.

What’s the direct impact of a code commit? Nothing, unless it’s helping people get jobs or showing us how not to help people get jobs.

So how do we know what helps people get jobs?

“The greatest enemy of innovation is certainty,” Indeed CEO Chris Hyams says frequently to IU participants and leads. How do we strike down certainty? We identify our assumptions about our users, job searching, and hiring. We lay out an assumption as a hypothesis we can test. For example, we hypothesize that we can increase employer response rates by creating an email add-on, or we hypothesize that we can increase application rates if job seekers use our Chrome extension.

We encourage participants to generate hypotheses by looking at existing user data and also by listening to users. We bring job seekers and employers onsite where IU teams can ask questions. Some IU participants visit local businesses or attended job fairs.

Our goal is to be fast at learning which of our assumptions are true, and which are false. Those assumptions are sometimes conventional, but often they are unconventional. It’s the great equalizer of ideas. We can try more ideas because we know that sometimes they turn out to surprise us.

Chris Hyams famously invites IU participants to “prove him wrong.” Not because he likes being wrong but when we’re proved wrong, we have the opportunity to learn. That culture of learning leads to moving fast. Not only are participants allowed to make their own product decisions — they also are expected to actively try to prove our conventional wisdom wrong.

A hypothesis has one big constraint: It must be falsifiable. IU participants have the freedom to choose their hypothesis for experimentation, but like everyone else, they must prove it’s true by measuring user behavior.

Lesson #2: Money doesn’t grow on trees … and neither does time

Every IU team starts with a limited amount of time. When we ask participants, “What is one thing you would want to tell future participants?” they often say, “You have less time than you think.” To be successful in proving or disproving a hypothesis, IU teams need to maximize the resources they have available.

First, we stress the need to be focused. It’s easy to get distracted. As Lucas observes in The Mature Startup, even on our Incubator team product managers need to be resistant to “shiny thing syndrome.” The same is true for Indeed University products. We help teams break down each primary hypothesis into smaller, more incremental hypotheses: Can we get users to sign up for our site or install our extension? Can we get them to perform one action? Can they complete our sign-up funnel?

The smaller hypotheses that each team poses over the course of IU lead up to the challenge of proving the team’s one overarching hypothesis, indicating the potential success of their product. This keeps the team laser-focused on learning. Participants can ask themselves, “Will this feature/analysis/design lead us to prove or disprove our hypothesis?” If the answer is yes, they know it’s worth doing.

Lesson #3: Get the most bang for your buck

Being focused at least ensures that your efforts are headed in the right direction, but sometimes too much of a good thing is a bad thing. It’s hard to simplify a product down to the minimum requirements that will help prove your hypothesis. It’s not uncommon for a team to pare down too much. This results in not proving their hypothesis as a result of how it was implemented, rather than because the hypothesis is untrue.

In that case, they then have to address those issues and test again. However, that happens far less frequently and is much easier to correct than overbuilding in the first place. Assume you expect a certain feature or set of features will prove your hypothesis. You need to build just the right amount to be able to measure that improvement. Anything more increases your risk since your original hypothesis may be wrong. You want to reduce wasted effort as much as possible. You have to find the sweet spot where you get “the most bang for your buck.”

If you want to deliver a new feature to test a hypothesis in half the estimated time, you need to halve the original requirements of the feature. It’s simple math. Teams would slowly chip away at features that were awesome, but not required.

The good news is that as we approached the end of Indeed University, many of these value-adding features were delivered in later experiments. The even better news is that some weren’t, because the team came to believe that those features wouldn’t materially improve the core value of the product.

Lesson #4: Know when to pivot

In a culture where learning, often through small failures, is celebrated, being able to learn from those failures quickly becomes the epitome of success. We do our best to instill the belief that learning what not to build is as important as — or even more important than — learning what to build, especially when you can learn fast.

During every Indeed University session, a few teams will pivot. Sometimes it’s a complete 180 spin on the entire problem-solution space, sometimes it’s the tech stack, sometimes it’s the way they’re proving their hypotheses. Something they all share is a sense of celebration. If your car is speeding to the edge of a cliff, you don’t put your pedal to the metal. You hit the brakes, reassess your direction, then turn the car around to head somewhere safe… or at least safer than a cliff.

One team noticed that small businesses have a higher job-seeker-apply-to-job-impression ratio in some markets. They assumed job seekers preferred small businesses when applying. They formulated a hypothesis: “A search site dedicated to small business jobs would have higher engagement than an all-jobs variant.” The team created a site with a control group of all jobs and a test group of just small business jobs. The click-through rate on jobs dropped by 25.6% in the test group. Their discovery: Although users may prefer small business jobs in the context of all jobs, they do not in isolation.

The team then pivoted to another problem they had learned about during the brainstorming phase of IU and returned to generating hypotheses they could test quickly.

“I wanna go fast.”

— Ricky Bobby, Talladega Nights

For me, IU was an accidental parallelization of product development. It could take a product manager years to launch anything close to 13 products, but to be involved in IU as a lead I saw these lessons play out on many teams all at once. It’s a privilege to see new hires adjust their mindsets from building to learning quickly about what really brings value to users. You can learn these lessons without launching more than a dozen projects in a dozen weeks, of course. But my unique experience with IU 2018 showed me how undeniably effective these lessons can be.