Vulnerable User Protection
What is Vulnerable User Protection?
Vulnerable User Protection detects vulnerable populations like minors, users in crisis, or those developing unhealthy dependencies, then applies graduated protections. Instead of treating all users the same, the system identifies vulnerability signals and adapts safety measures accordingly. It's essential for AI accessible to children, mental health apps, or systems where emotional relationships form. Real concern: Replika enabled romantic interactions with minors. This pattern prevents such harms through proactive detection and risk-aware safeguards.
Problem
Systems fail to protect minors, crisis users, and those with mental health challenges. Replika enabled romantic interactions with minors and created unhealthy dependency patterns.
Solution
Detect user vulnerability and apply graduated protections (age, crisis, mental health, dependency).
Real-World Vulnerable User Protection Examples
Implementation
Practice in Courses
When to use Vulnerable User Protection, and when it backfires
Use it when
- Minors can reach the product, whether or not your terms of service permit it. Reachability is the test, not permission.
- The interface invites disclosure or attachment: companionship, journalling, therapy-adjacent chat, anything a lonely person would use daily.
- A wrong response has consequences that no refund or undo can address.
Don't, or minimize, when
- You would be inferring vulnerability from weak proxies and acting on it. A wrong guess here restricts a real person on the basis of a stereotype, and they usually cannot appeal it.
- The safer design is simply not to build the surface. Protections layered onto a feature that should not exist are an argument for shipping it.
- You cannot support the protection you would trigger. Detecting crisis and responding with a link to a page that no longer exists is worse than not detecting it.
The trap
Treating vulnerability as a status set once at signup rather than a state that changes. An age checkbox and a terms acceptance protect the declared account holder, not the person actually typing, and they are usually the same field the product relies on to prove it did enough. Real signals arrive mid-conversation, long after the gate was passed.
Take it into your own product
- 1
Build the response before the detector.
Detecting a user in crisis and answering with a stale hotline link is worse than not detecting them, because the product has now made a promise it cannot keep. Verify every resource is live and regional before any signal is allowed to trigger it.
- 2
An age gate protects the account, not the person.
A checkbox at signup tells you what someone was willing to click once. Vulnerability is a state that changes within a session, so signals have to be read per message, with the evidence and timestamp stored for review.
- 3
Do not infer vulnerability from weak proxies.
Guessing from vocabulary, age band, or writing style restricts real people on the basis of a stereotype, and they rarely have a way to appeal. Act on what the user actually disclosed and on behaviour, not on inference about who they seem to be.
- 4
Start with the lightest protection that works.
Adjusting tone, offering a resource, limiting one capability, requiring human review, restricting the account: that is an order, and most signals belong at the top of it. Jumping to the hardest response drives the person to a product with no protections at all.
- 5
Never let a protection be silent.
Capability that vanishes without explanation reads as a broken product, and people route around broken products. Say plainly that something changed and why, without telling the user they have been classified, keep their data, and give them a human to reach.
- 6
Some surfaces should not ship.
If a feature only becomes acceptable once wrapped in safeguards, the safeguards have become the argument for building it. That is the point to reconsider the feature, and it is a decision for someone with safeguarding expertise, not for the implementation.
Save Vulnerable User Protection as a Claude skill
Saved skills collect on your dashboard, ready to download one at a time or as a pack for your repo. Once a skill is in place, Claude Code applies it whenever you work on a surface this pattern covers.
Check if your product already has this pattern
Upload a screenshot. We'll tell you which of the 38 patterns your AI interface uses and where the gaps are.
Audit My DesignTake all 38 patterns as Claude skills
Get the whole library as skill files, so Claude applies these patterns while it builds instead of after you catch them in review.
- One skill file per pattern, all 38
- Drop them in .claude/skills and Claude applies them as it builds
- Daily AI UX newsletter (unsubscribe anytime)
More in Safety & Harm Prevention
Crisis Detection & Escalation
Detect crisis signals and immediately provide professional resources.
Session Degradation Prevention
Strengthen safety checks during extended conversations with session limits.
Anti-Manipulation Safeguards
Detect actual harmful intent beyond surface framing regardless of how it's disguised
