I stole this idea, cause it was so good…
At Smule I worked alongside Kevin Sung, Kevin is a stellar PM and one of the sharpest people I’ve ever met. He had worked at Zynga, and learned the concept of the walkthrough there, which he spread to the other PMs at Smule. I immediately saw it’s immense value, especially before shipping major changes or new features.
Here, I’ll explain my take on the context, process, and benefits of a team product walkthrough, specifically for mobile devices, but really applicable to any complex software.
What you need
There are a few things your team and product will need to make the most of a walkthrough:
- A set of user stories in an outline, or workflows for the features or product to be analyzed. If you have data which tells you common user flows and funnels, consider using those to write and structure your outline.
- A pre-release build.
- A collaborative note-taking document (at the Wikimedia Foundation we use something called Etherpad), but google docs and many other things will do.
- The ability to steam a device screen (and ideally record) in a way the whole team can see.
- A healthy team dynamic, and a team that can deliver and accept critique professionally.
The basic process
Gather the full development team for a product or feature in a room (physical or online). Appoint one person as “the finger”. This person will hold the device, and act as the finger of the platonic user or persona. This can be any person on the team. The device should be streamed (if the team is remote) or mirrored (for a room) and also ideally recorded for later review. Additionally a note monitor should be appointed. This person will write the PM and finger’s notes (as needed), and ask the PM to stop narrating if note taking is lagging behind the walkthrough pace.
The PM should begin to narrate the user’s basic motivations and desires, walking through the user story/ies to be analyzed. The finger is then responsible for actually finding and taking the desired action.
As this is happening the rest of the team are responsible for watching the stream and looking for issues. Notes and issues should not be discussed out loud (only the PM, note monitor and the finger should be speaking, by and large), but rather noted in the shared document. On the other hand ALL written comments and notes should be encouraged, be they bugs, useability notes, visual issues, brand standards, anything. Each function (UX, frontend engineering, QA, marketing, etc) should especially focus on their area, and attempt to look with fresh “user” eyes at their team’s work for the first time again.
At the end of this process, each team member should summarize for the team, their thoughts on the state of the app as they just saw it. Is it ready for release? Did they see any potential blockers? How do they feel about their work?
After this the PM, potentially in combination with team leads (depending on team size and product complexity), should look at each note in the doc and triage it. I usually keep it simple and do four buckets: blocker, follow-up, punt, ignore.
- Blocker: the product cannot be released until this is fixed or remedied. I strongly recommend you file these immediately into your task management flow, to show the they are the most urgent, and came directly from the team’s efforts at the walkthrough.
- Follow-up: the comment, note or issue seems relevant, but is too vague to be actionable. I usually turn these into personal to-dos, send a follow-up email, or add them to future standing meeting agendas (planning, standup, 1:1, etc).
- Punt: the issue is valid but not a blocker or requiring immediate action. File and forget.
- Ignore: the note is not valid, actionable, etc.
Key benefits and points
Besides the direct benefit of the issues found and improvements that result, this process has a number of other benefits for the product and the team:
- User oriented: The whole process reinforces the primacy of a user and their story, increasing user empathy, and encouraging the team to think of their work as a whole experience.
- Quality reinforcing: quality is everyone’s job, and everyone has their own taste. This process focuses everyone on the quality of what they are about to put out, but allows everyone to examine and express their opinion in their own area. It also provides a way for introverts and people who are better in writing to give feedback about the product.
- You do it together: The whole team that delivers the product should participate. The line engineers, the ops guys, customer service, the marketing folks, anyone that shapes or needs to know this product or feature should participate. This encourages a sense of collective ownership and the summary reviews can be excellent ways for functions that may not normally meet to hear directly from each other.
- Reality-based: like writing a term paper, it’s never a good idea to review your own work. Looking at each other’s work brings a dose of reality to the editing and iterating process. False impressions and misunderstandings clear up quickly in the light of a shared experience.
- Follow-through: I cannot stress how important it is follow up on all legitimate comments and notes, especially the first few times through the process. If the team thinks they’re listening to you narrate the world’s boringest livestream, they will disengage and the time will be wasted. Consider their feedback seriously and take action where needed.
- As Jeff Smith, CEO of Smule once (justifiably) told at a PM: “I don’t care what the numbers say. USE THE FUCKING PRODUCT”. This process forces everyone to see what it’s like to use the fucking product. That can only be good.