Free Cheat Sheet to Document Your Experiments
How to foster team learning by documenting your decisions
Running experiments is useless…
… unless you learn from them in a structured way!
Product teams often invest a lot of time in their experiments (which is great), but pay much less attention to documenting their findings. Which is a pity!
If you don’t document your experiments properly
❌ new hires have no clue what you have tested in the past and might end up testing the same hypotheses again
❌ you are less likely to spot flaws in your validation setup (it really helps to think about how you will document your learnings BEFORE you run the experiment!)
❌ you are depriving your team of an opportunity to reflect not only on the decision at stake, but also on the meta learnings on how to run future experiments
Documenting decisions in a structured way
Here is a little cheat sheet on how we document decisions at Product Academy.
Decision at stake: This is the most important part of the documentation - and yet it tends to be overlooked or very fuzzy. Yes, it’s great that we are now able to run lean experiments. However, if they are not helping you take decisions in a structured way with increased confidence, they are still a waste of time! So before you get all excited about running your next fake door test or user interviews, write down exactly which decision(s) you are trying to facilitate.
Hypothesis: Crafting an actionable hypothesis is part art and part science. Wording matters. Uncovering underlying assumptions is crucial. So is being clear about assumed correlations and how strong or weak they are. If you feel you could need some help improving your hypotheses, check out our hands-on online workshop.
Experiment Setup: Did you incentivize your test users? Who exactly knew about the test landing page? All website users? Just your existing customers? It is important to document how experiment results have been gathered in order to not misinterpret them.
Key Findings: How do you interpret the data and signals you gathered? Which learnings are you deriving from your experiment?
Decision Taken: Now it’s time to tie everything back to the decision at stake. What are you going to do based on the experiment results?
Responsible Team: Who was involved in running this experiment? It’s always helpful to know who you can reach out to in case of questions.
Meta Learnings: This is my favorite part! What can you take away from this experiment on a meta level? Id est: What did this experiment teach you about how to improve future experiments in general? Maybe your hypothesis was not specific enough. Or your test group turned out to not be representative of your average customer. Maybe you realized that next time you would like to have one person conducting the interview and another one taking notes. Or it was a fabulous experience to invite some of your developers to witness this user testing. Take a moment to reflect on the meta learnings - this will pay off big time!
Download our Miro Board for free
Here is the link to our free Miro Board. Knock yourself out and let us know if this was helpful!
Thanks for reading Tanja's Product Academy Digest! Subscribe for free to receive new posts and support my work.