Aleodor
Kickstart01 · week two · Friday
Reading the evidence
What your users did and said, what it means, and what you change on Monday.
With Alexandru Daicu and Aleodor Tabarcea.
Aleodor
What the two hours are for
Tonight we read your notes, not your idea.
Nobody is here to decide whether your thing is any good. We are here to work out what your users actually told you, which is rarely the same as what you heard.
Aleodor
Before anything else
You got fewer than you planned.
Everybody does, every time, and we did too.
The ones who said no told you something as well. Nobody ever writes that part down.
Both
Where we are
The middle of this picture is behind you now. Tonight is the right-hand side.
Alexandru
Part one
What happened, before what it means
Alexandru
The reality check
Everyone wants to start at the fix.
Hold off for an hour.
Every minute spent on the solution before anyone agrees on what actually happened is a minute spent decorating a guess.
Alexandru
What is in your notes, and what you added
What a camera caught
"Scrolled past the profile step twice."
"Said: I do not know what to put here."
"Opened another tab for forty seconds."
What you added afterwards
"She was confused." That is a theory.
"The form is too long." That is a conclusion.
"We should drop the profile step." That is a decision.
Only the left column is evidence. The right-hand one drifts a little further from the session with every line.
Alexandru
Your notes, now
Split your notes into two columns
- Left: only what a camera would have caught. Actions, and their words.
- Right: everything you added. Keep it, it is useful, just move it.
- Circle every sentence you have word for word. Those are your best material.
Any line with "confused", "frustrated" or "obviously" in it belongs on the right.
Alexandru
What you are about to find
Your left column is going to look thin.
Most of them do, the first time round. Good.
You have just worked out what to write down, which is why next week you will come back with twice as much.
Aleodor
Part two
One thing you saw, more than one explanation
Aleodor
She typed "a technical guy, I guess"
Three explanations fit. They lead to three different products, and the sentence itself will not tell you which one you are holding.
Aleodor
The test of a good interpretation
Four questions to run on every conclusion
- What did you actually see that made you say this?
- What is another explanation that fits the same thing?
- What would tell the two of them apart?
- How many people did this, out of how many you watched?
If you cannot name a second explanation, you did not interpret anything. You picked one.
Aleodor
In your breakout room
Take somebody else's conclusion apart
- Read your right-hand column out to the other team in your breakout.
- For every line in their right-hand column, ask what else would explain it.
- Write down the alternative they give you beside each one.
You are not there to be nice about it. You are there to find the second explanation.
Alexandru
Part three
What they said, and what they did
Alexandru
When the words and the behaviour disagree, believe the behaviour.
Nobody lied to you. People describe the version of themselves they would like to be true, and then go home and do exactly what they always do.
Alexandru
I would definitely use this.
The most dangerous sentence anybody brought back this week.
Alexandru
The same session, two kinds of material
What they said
"I would definitely use this."
"This looks really clean."
"Something like this is badly needed."
"I will tell my friends about it."
What they did
Asked when it ships.
Tried to add somebody who is not on it yet.
Sent you a name, unprompted.
Closed the tab after twelve seconds.
Nothing on the left is a lie. Nothing on it is evidence either.
Alexandru
The reality check
The ones who said no are data too.
You asked ten, three said yes. That ratio is about demand, and it is nowhere on your sheet.
Alexandru
Break
Five minutes
Alexandru
Part four
The value verdict
Alexandru
Did anything happen that would not have happened if they did not care?
An introduction. A calendar slot. A document. Money. A question about when it ships.
Something it cost them to do.
Alexandru
What counts, and what does not
Three signals, and three that look like signals
They gave you a name, a slot, a document, or money.
They asked something you did not set up. When does it ship, can I add someone.
More than one person, in different situations, did the same thing.
"Looks great." "Really needed." "I would use this."
"If it had X, I would definitely..." A sentence about a future that has not happened.
The part where you explained it well and they agreed with you.
Alexandru
Your value verdict
One sentence, then one of three words
- The sentence: what happened that would not have happened if they did not care.
- Then one word. Yes, no, or we did not find out.
- If it is the third one, write the question you will ask next time.
Note that "promising" is not one of the three words.
Aleodor
Part five
The usability verdict
Aleodor
Count. Do not narrate.
"Four of the five stopped at the same step" is a finding.
"People found it confusing" is a mood.
Aleodor
For every step you watched
Three numbers, for each step
- How many reached it.
- How many got through it without help.
- How many needed you to say something.
The third one is the honest number.
Aleodor
Your usability verdict
Write the numbers, step by step
- One line per step: reached, finished alone, needed help.
- Mark the step with the worst third number.
- Beside it, one sentence on what you saw there, from the left column only.
Denominator every time. Three out of three and three out of twelve are different sentences.
Both
Part six
Two problems, two different fixes
Both
Put your own verdict on this grid
The two halves of this grid do not respond to the same kind of week.
Both
A usability problem and a value problem are not the same problem
If they could not work it
Fix the step. It is usually small.
Retest that same step with three people.
You keep your plan.
If they did not care
Changing the button will not help.
The next move is a different user, a different need, or a different thing.
You change your plan this week, while changing it is still cheap.
The most expensive thing anyone here can do this week is polish a button on something nobody asked for.
Both
The reality check
Nobody wants it? Good week.
It cost you four days. The other version of this conversation happens in a year, with a team, a runway and considerably more people in the room.
Alexandru
Part seven
What you change, and what you test next
Alexandru
Sort every finding by two things
How sure are you?
More than one person, from the left column, in different situations.
Or one person, once, and you liked what it implied.
What does the fix cost?
An afternoon.
Or a rebuild.
Cheap and certain first. Never start a week on the expensive thing you are least sure about.
Alexandru
Three columns, nothing else
Now, next, not now
- Now. You are sure, and it is cheap. It happens this week.
- Next. You are not sure yet, so it becomes a question in the next test, not a feature.
- Not now. Everything else, with a date you will look at it again.
A list with an empty third column is a wish, not a plan.
Alexandru
On your sheet
Write your three columns
- Three things in "now", at most. Fewer is better.
- One question in "next". The one you most need answered.
- Everything else in "not now", with a date.
Then underline the question. That is next week's test, already half written.
Aleodor
What you keep from this week
One question. One answer.
That is all a test ever is. You asked, you got an answer, and you now have a sharper question than the one you started the week with.
Both
Same sheet, next question.
It works again next week, and the week after that. The only thing that changes is what you put in boxes three and four.