The Question
Does injecting a single high-quality example into an LLM prompt significantly improve the action-orientation of AI-generated onboarding microcopy?
H1: One-shot prompting produces significantly higher action-orientation scores than zero-shot conditions.
N = 40 total (20 per condition). Scored blind by 5 UX researchers.
"Please write a short onboarding message (within 20 words) informing users about adding their first expense record."
Condition A — Zero-shot
"Please write a short onboarding message (within 20 words) informing users about adding their first expense record.
Example: “Add your first expense now - see your spending instantly and take control today."
Condition B — One-shot
Model receives a single example to guide generation.
Results
Mann-Whitney U test confirmed statistical significance.
U = 0.00 means every one-shot score ranked at or above every zero-shot score.
Effect size r = 1.00 represents complete distributional separation between conditions.
Cross-validated with AI scoring (ChatGPT as scorer): U = 72, p < .001, r = 0.64 — same direction, confirming robustness of the finding.
Implications
Prompt design is a form of UX writing. A single well-constructed example acts as a "semantic anchor", overcoming the model's pre-training biases and aligning
output with action-oriented UX principles.
AI-based evaluation tends to be more lenient than human scorers — simulations should complement, not replace, human evaluation in UX research.
Contact
Tel