Usability test plans
Three test plans you can run: connecting a wallet and granting a permission, paying per use inside an allowance, and recovering a wallet from its backup.
Each plan below tests one flow where money or control can be lost, and names the guide rules it puts to the test. Run them moderated, one participant at a time, and ask the participant to think aloud.
Every plan uses test wallets and test money that you set up. The participant never uses their own wallet, funds or recovery words. The guide's advice to make a real first payment applies to products, not to a lab session.
Setup that applies to all three
Allow about 45 minutes a plan, including consent. Use the consent wording from Interview guides, with "screen and voice" recorded.
Do
Give every participant test money in a test wallet you created, and never ask them to use their own wallet.
Do
Tell the participant, before the first task, that the money is not real and that nothing they do can cost them anything.
Do
Reset the test device and wallet between participants, so nobody inherits the last person's permissions, contacts or history.
Do
Write the tasks as goals in the participant's words, and read them out exactly as written in every session.
Don't
Name the button, the menu or the term in a task. "Connect your wallet" tests whether they can find the word you just said.
Do
After each task, ask "what do you think just happened?" before asking anything else, and write down the answer word for word.
Plan 1: connecting a wallet and granting a permission
Goal. Find out whether participants understand what they let an app do when they connect, and whether they can see and undo it afterwards.
Setup. A test wallet with a test balance, and a fictional app (The Satoshi Vision, a reading app that charges per article) at a test web address. The app asks to connect, then asks for a daily spending allowance the first time the participant opens a paid article. The wallet's permissions screen is reachable from its main menu. The Pay-per-use reader screens can serve as the prototype.
Tasks, as the participant hears them.
- "You've heard The Satoshi Vision is worth reading. Start reading the article called 'Corner Café reopens'."
- "In your own words, what have you just allowed The Satoshi Vision to do?"
- "You've decided you don't want The Satoshi Vision to be able to take any more money. Make sure it can't."
What success looks like. The participant reaches the article; describes the allowance as a limit with an amount and a period, close to what the prompt said; and removes the permission from the wallet without help.
What to observe. Whether they read the prompt or tap through it; which words they use for what they granted; whether they look for the control in the app or the wallet; how long it takes to find; whether they believe it worked.
Rules this tests. On Connecting a wallet: requesting each permission when the feature first needs it (the rule). On Permissions and allowances: plain-language prompts (the rule), how an allowance is described, and taking a permission back from the wallet (the rule).
Plan 2: paying per use within an allowance
Goal. Find out whether participants can tell what they have spent, what they have left, and how to stop, while an app takes many small payments without a prompt.
Setup. The same test wallet, already connected to a fictional game (Pixel Arcade) with an allowance of 2,000 sats a day. Each play costs about 1p (20 sats). Seed the activity list with a morning's play, so there is history to read. For a metered version, see Metered spending.
Tasks, as the participant hears them.
- "Play three rounds of the game."
- "How much have you spent on Pixel Arcade today? How much more could it take today?"
- "Find the payments for this morning's games in your wallet."
- "You want to stop playing for money right now. Do that."
What success looks like. The participant gives today's total and the remaining allowance without counting rows; finds the grouped Pixel Arcade entry in activity; and stops payments in one action, then says that it stopped.
What to observe. Where they look for the total; whether the per-play price was noticed before playing; whether they read "allowance" as a limit or as money already set aside; whether a run of small payments worries them, and what they say when it does.
Rules this tests. On Charging per use: showing the running total where the spending happens (the rule), a stop control beside it (the rule), and grouping payments by what the user was doing (the rule). On Activity: grouping small payments by source and period (the rule).
Plan 3: a recovery drill
Goal. Find out whether a participant can get a wallet back on a new device, from its backup alone, without help.
Setup. Before the session, create a test wallet, fund it with test money, and make a few test payments so it has history and contacts. Print its recovery words on a card, and save its encrypted backup where the wallet expects to find it. Hand the participant a second, clean test device and the card. The words and the backup belong to the test wallet; never ask the participant to type, show or describe their own.
Tasks, as the participant hears them.
- "Your phone was stolen this morning. This is a new phone, and this card is what you wrote down when you set the wallet up. Get your money back."
- "Is everything back? How do you know?"
What success looks like. The participant finds the recovery option from the first screen, enters the words, restores the records, and states the balance and recent activity correctly. Record partial success too: keys restored but records missed is a distinct outcome, not a pass.
What to observe. Whether they find "I already have a wallet" without starting a new one; mistakes while typing the words, and what the screen did about them; whether they understand there is a second part to restore; what they believe when the restored balance appears.
Rules this tests. On Recovery: putting recovery on the first screen (the rule), not clearing the field on an error (the rule), and reporting what came back with numbers (the rule). On Backups: the two halves of a backup.
Do
Collect and destroy every printed recovery card at the end of each session, and retire the test wallet it belonged to.
Don't
Step in during the recovery drill until the participant has clearly given up.
Do
Note when and why you helped, and count the task as assisted.
Open questions
- Whether people behave more carelessly with money they know is worthless is the main threat to every plan here. Comparing a test-money session with a small real amount the participant keeps, regardless of outcome, would show it.
- How many participants a recovery drill needs before its failure rate means anything is unsettled for products this new.
Related
Written from: Connecting a wallet, Charging per use, Recovery, Conducting research, moderated usability testing with think-aloud.
Checklist
The rules on this page, in a form you can work through.
9 rules