Module 4: EU AI Act Deep Dive

2. Unacceptable Risk: Prohibited Systems

Next →

Capsule description

The EU AI Act's strictest category: Unacceptable Risk. These systems are prohibited. It isn't "high oversight" or "extra requirements" — it's you cannot deploy them in the EU. Period.

The line is clear, but the specific criteria demand attention: bona-fide use cases can accidentally trigger a prohibition if you don't understand the exact criteria.

This capsule covers the 8 categories of unacceptable risk, with concrete examples, qualification criteria, and the limited exceptions. By the end, you'll be able to identify whether your system potentially violates these prohibitions, and avoid it.

Stakes: fines up to €35M or 7% of global revenue. The most sanctionable category. Take it seriously.

The 8 categories of unacceptable risk

EU AI Act Art. 5 enumerates the prohibited practices. The 8 categories:

1. Subliminal manipulation causing harm

The Act's text: "AI systems that deploy subliminal techniques beyond a person's consciousness, or purposefully manipulative or deceptive techniques, with the objective or the effect of materially distorting the behaviour of a person or a group of persons by appreciably impairing their ability to make an informed decision."

Technical translation:

  • The system uses techniques the user doesn't consciously perceive.
  • It causes the user to make decisions different from those they'd make consciously.
  • Those decisions cause material harm (financial, psychological, physical).

Prohibited examples:

  • Subliminal audio that induces impulse purchases.
  • Visual patterns that create compulsive addictive behavior.
  • Extreme personalization that exploits psychological vulnerabilities (targeting gambling addicts).

Cases NOT prohibited:

  • Standard marketing personalization (acceptable).
  • A/B testing to optimize UX (acceptable).
  • Algorithms that show content the user is more likely to engage with (general recommendation, NOT subliminal).

The line: subliminal causing material harm. Both criteria are needed for the prohibition.

2. Exploitation of vulnerabilities

Systems that exploit vulnerabilities based on:

  • Age (especially minors).
  • Disability.
  • A specific social or economic situation.

In order to distort behavior causing material or psychological harm.

Prohibited examples:

  • An AI toy that manipulates children into asking their parents for purchases.
  • A system targeting elderly people with cognitive decline for predatory loans.
  • Targeting addicts with personalized content that feeds the addiction.

Critical: targeting vulnerable groups + harm is the combination that triggers the prohibition.

3. Social Scoring (government and private companies)

General-purpose social scoring by public authorities or private actors:

  • Evaluating or classifying people based on social behavior, personality traits, or predicted characteristics.
  • Resulting in detrimental or unfavorable treatment in unrelated contexts.
  • Or treatment disproportionate to the social behavior.

Prohibited examples:

  • A China-style social credit system (government).
  • A company rating customers based on general behavior across services in order to deny services.
  • A general-purpose "trustworthiness score" system.

Exceptions:

  • Sector-specific scoring (credit scoring only for credit) — not unacceptable, but high-risk.
  • Legitimate business assessments within a specific context.

The line: scoring that's comprehensive across contexts and produces disproportionate treatment.

4. Predictive policing based solely on profiling

Risk assessment of people to predict the probability of committing a crime, based solely on profiling (personality traits, etc.).

Exceptions:

  • When it's applied to support human assessment (not predictive policing on its own).
  • Based on objective verifiable facts (not profiling).

Prohibited examples:

  • An algorithm predicting future criminality based on demographics, neighborhood.
  • A system scoring "likelihood of recidivism" based on profiling alone.

Legitimate exceptions:

  • Risk assessment based on specific evidence and human review.
  • Forensic analysis for a specific investigation.

Critical: profiling alone is the problem. Combined with human review and objective evidence, it can be acceptable (high-risk, not unacceptable).

5. Untargeted scraping of facial recognition databases

Creating or expanding facial recognition databases through untargeted scraping of:

  • Facial images from the internet.
  • CCTV footage.

A real prohibited case: Clearview AI's business model — scraping social media to build a facial recognition database of billions of people without consent.

Exceptions:

  • Targeted use with a specific legal basis (e.g., investigating a specific crime).

6. Emotion recognition in the workplace and education

Inferring emotions of natural persons in:

  • The workplace.
  • Educational institutions.

Exceptions:

  • Medical use (emotion recognition for health).
  • Safety reasons (e.g., driver fatigue detection — debatable).

Prohibited cases:

  • A system monitoring employees' emotional state during work.
  • A system evaluating students' emotional engagement during class.

The reason: the workplace and education are contexts where a power imbalance prevents free consent. Emotion recognition here is coercive.

7. Biometric categorization for sensitive attributes

Biometric categorization to deduce:

  • Race, political opinions, religion.
  • Trade union membership.
  • Sex life or sexual orientation.

Exceptions:

  • Authorized labelling for datasets (with safeguards).
  • Law enforcement (specific limited cases).

Prohibited cases:

  • A system classifying people by race based on biometric features.
  • A system deducing sexual orientation from voice features.

The reason: these attributes are specially protected under the EU human rights framework. Inferring them biometrically is a violation of privacy and dignity.

8. Real-time biometric identification in public spaces

Real-time remote biometric identification by law enforcement in public spaces.

Exceptions (strict):

  • A targeted search for specific victims (kidnapping, missing children).
  • Prevention of a specific imminent threat (an imminent terrorist attack).
  • Locating or identifying suspects of specific serious crimes.

Requirements for the exceptions:

  • Prior judicial authorization (with limited emergency exceptions).
  • Targeted (no mass surveillance).
  • Time-limited.
  • Documented.

The main reason: mass surveillance via real-time biometric ID has a chilling effect on fundamental freedoms.

How to avoid accidentally landing in these categories

Even bona-fide use cases can trigger prohibitions. The defenses:

Defense 1: Document intent and context clearly

If your system is legitimate:

  • Document what specific purpose it serves.
  • Document who specifically will use it.
  • Document what safeguards prevent misuse.

Documentation matters when a regulator asks.

Defense 2: Apply a proportionality test

For each feature of your system:

  • Does it serve a legitimate, specific purpose?
  • Is it proportionate to that purpose?
  • Are there alternatives that achieve the purpose with less risk?

If you can't justify each one with specifics, consider removing it.

Defense 3: Avoid problematic feature combinations

Combinations that trigger prohibitions:

  • Vulnerable group targeting + harm potential → the exploitation prohibition.
  • Comprehensive scoring + cross-context impact → the social scoring prohibition.
  • Profiling + automated criminal prediction → the predictive policing prohibition.
  • Biometrics + sensitive attribute inference → the biometric categorization prohibition.

If your system combines these, redesign it.

Defense 4: Implement human-in-the-loop

Many borderline cases move from "prohibited" to "high-risk" (still legal, just with more obligations) when human decision-making is preserved.

Example: predictive policing alone = prohibited. Predictive policing as an input to a human decision + objective evidence = high-risk (acceptable).

Defense 5: Geographical scope

If your system could trigger a prohibition but you only deploy in non-EU markets, that's OK from the EU AI Act's perspective (other regulation may apply in those markets).

But: geofence properly. EU users accessing it = the scope applies.

What to do if your system is in the prohibited category

An uncomfortable honesty: if you analyze your system and land in unacceptable, the options are:

Option 1: Re-architect

You can often eliminate the problematic component without losing core functionality:

  • Remove subliminal manipulation features → you keep the service.
  • Remove vulnerable group targeting → a broadly applicable service.
  • Add human review to predictive policing → high-risk (acceptable).
  • Limit the scope of biometrics to sector-specific → high-risk.

Re-architecting costs engineering time but preserves the business.

Option 2: Geographic exclusion

If re-architecture is impossible, exclude EU users:

  • Geofencing.
  • Account creation restrictions.
  • Terms of service preventing EU use.

Trade-off: you lose the EU market. Acceptable for some companies, existential for others.

Option 3: Don't build it

Sometimes the answer is: this product shouldn't exist as designed. The market may not support it elsewhere either, given the growing global regulatory alignment.

That's a harder business decision. Sometimes it's the right one.

Common traps

1. "My system is 'inspired by' but not exactly..."

Regulators look at substance, not marketing. If the function matches a prohibited category, the prohibition applies regardless of how you describe it.

2. Underestimating "manipulation"

Many recommendation engines walk a fine line. If you exploit cognitive biases to drive specific behavior, especially in vulnerable groups, consider the exploitation prohibition.

3. "We have terms of service preventing this"

Terms of service ≠ technical safeguards. If users can misuse your system in prohibited ways, you may be liable regardless of what the TOS says.

4. Treating the "real-time" definition loosely

Real-time biometric identification includes near-real-time. A 1-minute delay is still "real-time" for regulatory purposes. Don't think you can dodge it by adding a small delay.

5. Assuming the law enforcement exception applies broadly

The exceptions for law enforcement biometric identification are very narrow. Most use cases don't qualify. Default: assume the prohibition applies unless you're explicitly within the narrow exception.

Self-check

1. What's the difference between social scoring (prohibited) and credit scoring (high-risk)?

Social scoring (prohibited):

  • General purpose: it evaluates behavior across multiple unrelated contexts.
  • Cross-context impact: scoring in context A causes treatment in context B (a totally distinct one).
  • Disproportionate: the treatment is disproportionate to the behavior.

Example: a government rates a citizen based on shopping habits, social media, job performance, and friends — and that score determines access to housing, transportation, employment.

Credit scoring (high-risk):

  • Sector-specific: only for credit decisions.
  • Single-context: financial behavior → financial decisions.
  • Proportionate: relevant data only (income, history, etc.).

Both involve scoring people. The differences: scope (general vs. sector-specific) and proportionality (cross-context vs. same-context).

Implication: legitimate sector-specific scoring (credit, insurance, employment) is high-risk (regulated, but allowed). Comprehensive cross-context scoring is prohibited.

2. How do you keep your chatbot's personalization from falling into "manipulation"?

Standard personalization is NOT prohibited. The line with manipulation:

Acceptable personalization:

  • Showing content the user is more likely to engage with.
  • Recommendations based on stated preferences.
  • A/B testing UX to optimize satisfaction.
  • Adjusting tone based on the user's style.

Potentially manipulative:

  • Exploiting cognitive biases (loss aversion, FOMO triggers).
  • Targeting psychological vulnerabilities (anxiety, addiction).
  • Subliminal techniques (hidden patterns affecting decisions).
  • Time-pressure or fear-based triggers that materially distort decisions.

The test: would the user, if fully aware of the mechanism, still consent to it?

If "yes" (the user understands the personalization is helping them) → acceptable. If "no" (the user would be uncomfortable knowing exactly how they're being influenced) → potentially manipulative.

Documentation helps:

  • Disclose the personalization features.
  • Avoid exploitation patterns.
  • Don't target vulnerable groups specifically.
  • Test with diverse users for unintended consequences.
3. Why is emotion recognition prohibited in the workplace but not in consumer apps?

The main reason: power imbalance.

In the workplace:

  • An employee can't freely refuse without job consequences.
  • Power asymmetry = no real consent.
  • Monitoring emotional state = coercive surveillance.
  • It could lead to discrimination based on emotional patterns.

In consumer apps:

  • The user has the choice to use it or not.
  • The power balance is more equal.
  • Consent can be more genuine.

The same logic applies in education:

  • A student can't freely refuse without academic consequences.
  • The power imbalance prevents real consent.

Where emotion recognition might be acceptable:

  • Medical (with consent and clinical justification).
  • Voluntary consumer apps (mental health support, with clear consent).
  • Safety (driver alertness, with limited scope).

The principle: contexts where consent is meaningful = potentially acceptable. Contexts with a power imbalance = prohibited or restricted.

Practical implication: HR analytics products that include emotion analysis must remove that feature or face the EU prohibition. A substantial impact on the HR-tech industry.

4. If my system is in a gray area, what do I do?

Practical steps:

  1. Document your interpretation:

    • Describe the system.
    • Identify which category(ies) potentially apply.
    • Argue your interpretation with evidence.
    • Note the alternative interpretations.

  2. Apply the conservative test:

    • Default: assume the stricter interpretation.
    • Comply with the stricter requirements.
    • Less exposure if the regulator reads it the stricter way.

  3. Get a legal opinion:

    • For high-stakes systems, get a formal legal review.
    • The documentation creates a record of due diligence.

  4. Engage with the regulator:

    • The EU AI Office accepts informal queries.
    • Regulatory sandbox programs may be available.
    • Better to ask than to guess wrong.

  5. Implement safeguards regardless:

    • Even if not technically required, implement strong safeguards.
    • It reduces real-world risk + future-proofs you against regulatory evolution.

  6. Re-evaluate periodically:

    • Regulator guidance will clarify the gray areas over time.
    • Re-check your classification annually.

What NOT to do: assume "they won't notice" or "we'll deal with it if asked." GDPR enforcement showed that regulators actively investigate. The same will happen with the AI Act.

Summary and next step

  • 8 categories of unacceptable risk, absolutely prohibited:

    1. Subliminal manipulation.
    2. Exploitation of vulnerabilities.
    3. General social scoring.
    4. Predictive policing based solely on profiling.
    5. Untargeted facial recognition scraping.
    6. Emotion recognition in the workplace/education.
    7. Biometric categorization for sensitive attributes.
    8. Real-time biometric ID in public spaces.

  • Bona-fide systems can accidentally land there: defend via documentation, proportionality, human-in-the-loop, geographic scope.

  • If your system is prohibited: re-architect, exclude geographically, or don't build it.

  • Penalties: up to €35M or 7% of global revenue.

Checkpoint: you should be able to identify whether your system potentially triggers any prohibition.

Bridge to the next capsule: capsule 03 covers High-Risk — the largest category and the one most likely to apply to your system. You'll learn exactly which use cases are high-risk, the qualification criteria, and how to identify whether your system is there.

Resources

  1. EU AI Act Art. 5 — Prohibited Practices — the text.
  2. Future of Life Institute — Prohibited Practices Analysis — analysis.
  3. Clearview AI Cases — a facial recognition case study.

Next: 03-high-risk-categories.md — High-Risk AI: categories and criteria.

Capsule 02 of 08 — Module 4 — AI Ethics & Compliance Guide

← Previous1. Module introduction: EU AI Act Deep DiveNext →3. High-Risk AI: Categories and Criteria