Analyst handbook

Prefix: I wrote this analyst handbook at Prezi during a really long breakfast on a silent Sunday morning in the summer of 2015. My idea was to share some best practices with junior analysts and new hires.

The handbook was quite popular within the company and all our analysts added their part, so it changed a lot over time. However, the text shared here is the original, that I wrote at that long breakfast. It has flaws and it doesn’t represent the current analyst handbook of Prezi.

In Prezi we work in cross-functional teams. These teams have a Product Manager (PM), UX Researcher, Designer, Developers and an Analyst.

Enjoy and feel free to comment and share.

This little article is about how to generate reliable insights, which can actually impact product decisions.

Let’s imagine a case when the product manager asks you an easy sounding question like: “How many users making more than 5 prezis?”

TL,DR: give estimation when you know what you will do. Iterate, communicate and re-estimate at every step if necessary, validate your numbers as much as you can.

The task

You got the question. Before starting to work on it or giving estimation when it will be ready:

Ask your PM to spend 10 minutes together on that question:

What is the decision she would like to make?
What is the user story behind the question?
Why does she think that an answer to the question will drive her decision?
What are the time contraints for the decision? Is it a blocker right now or it is an input for the next cycle’s plan?
A good test: if it is 10% of users what the decision will be? If it is 5%? 25%? (if she can’t answer this question the task may not be important)

After understanding the question give an estimation when will you have a plan on the research. If it is an important project it shouldn’t be more than a day. Don’t give estimation for the whole project before you have a plan.

The plan

Spend an hour on figuring out what the outcome will look like. Make the 2–5 charts you will show to the PM. What is the take-away from each chart?
What will the research table(s) look like? What time period will it cover? Will it have a time window for each user (aka cohort)? A research table is the one in between the logs and the final daily aggregations. For example first 30 days prezis by user.
Gather feedback on your plan 1: show it another analyst. Pro tip: if you can, ask the same analyst always. She will know your projects and product and that will help her to give you better feedback.
Gather feedback 2: show it your ux researcher team member, they have a better knowledge on how users work. She will give you good insights/new ideas.
Gather feedback 3: ask the analyst/data community if anybody has touched a similar project. You shouldn’t re-invent the wheel.
Check what you have in logs! Validate that you have data for everything you want to use in your research table. The best is to perform the events you want to measure and check the logs as they happen (live stream). If you don’t find something: ask your developers.
Prepare for a meeting with the PM: why do you want to this exact method? why did you choose this time window? Have a good estimation on how much time the whole project takes. Have good estimation on the deadline. Beware as time!=deadline. You have other things to do so open your calendar and find the spots when you are actually going to work on this particular research.
Meet the PM, explain and validate the plan. Will this actually drive decisions/actions? Update her on what other tasks you have and what are you priorities. Agree on a deadline. (Send a memo after if you feel like)

Execution

If at any point in time you face a blocker that might take more than an hour (logs are in worse shape than you expected, you got another more important task) don’t try to fix it. First communicate to the PM: this is the issue, this is my rough estimation on the delay. When you found the fix or workaround give a new estimation on when the project will be ready.

Gather all the logs for the time period you would like to check. Did we log these events throughout the whole period? Is there a missing day somewhere? Event number by day and distinct user number by day is a good start to check the quality of the logs.
Data compatibility: will the tables join easily? Check it: do we have the same amount of users after joining?
As soon as you have the charts give an update to the PM. Gathering feedback asap is really important.

QA

When you are ready with all your charts: validate your numbers.
How does this chart compare to others you made? How does it compare to something in our company KPI’s? Anything that similar in chartio?
If you have a big, slow query, refine it. Count the rows. Compare it with the original.
Show it to another analyst, your ux researcher, and your data community. “Do you understand the chart? What is the one sentence take-away from the chart? Do you think these numbers are accurate? How would you validate it?”

Present

Show all the charts together to your PM. Do it in a half hour meeting to make sure your have time explain everything in detail. It is important that she understands the charts but don’t stop there. What does that mean, what is the story behind? Talk about the decision need to be taken. Prepare for this meeting.
Ask for estimations on actions. This will be very important in the follow up phase.
Invite the uxer to the meeting above or have a separate one with her.
Show everything to your team in a daily follow up. Explain charts, tell the story and talk about the decisions/actions.

If it is something your team will need again: make automated charts. The majority of your researches should end up in automated charts. Add description to charts so it will be easy to understand it in two month to.
Take care of your code. Clean it up, comment, github.
Prepare a report email to the whole company.
Show the draft to the PM and to your analyst community. Make sure everybody understands what you would like to say.
Share the report with the company. Make sure all the people who should know about it reads it.
Reply to questions or state that you can’t reply and why not. Give estimation on when can you reply if you plan to reply later.
Enjoy your fame for 5 minutes.

Follow up

Be part of the team/sub-team that executes actions based on your research. Help them all through their way.
Measure the impact/outcome. If it is possible prepare for measurement in advance.
Make sure when the outcome is shared with the company it has charts and numbers to back it up. (instead of “it is released” it should be like: released to 5%, we estimated 10% increase and we got 12% increase.)