Módulo 4: EU AI Act Deep Dive

2. Unacceptable Risk: Sistemas Prohibidos

Siguiente →

Descripción de la cápsula

La categoría más estricta del EU AI Act: Unacceptable Risk. Estos sistemas están prohibidos. No es "high oversight" o "extra requirements" — es no podés deployarlos en EU. Punto.

La línea es clara, pero los criterios específicos requieren atención: bona-fide use cases pueden accidentalmente activar prohibition si no entendés los criteria exactos.

Esta cápsula cubre las 8 categorías de unacceptable risk, con ejemplos concretos, criterios de qualification, y excepciones limitadas. Al terminar, vas a poder identify si tu sistema potentially viola estas prohibitions y evitarlo.

Stakes: multas hasta €35M o 7% del revenue global. La categoría más sancionable. Tomarse en serio.

Las 8 categorías de unacceptable risk

EU AI Act Art. 5 enumera prohibited practices. Las 8 categorías:

1. Manipulación subliminal causando daño

Texto del Act: "Sistemas AI que despliegan técnicas subliminales más allá de la conciencia de una persona o técnicas manipuladoras o engañosas con el objetivo o efecto de distorsionar materialmente el comportamiento de una persona o grupo de personas perjudicando su capacidad de tomar decisiones informadas."

Traducción técnica:

  • Sistema usa techniques que el usuario no percibe conscientemente.
  • Causa el usuario tomar decisiones diferentes de las que tomaría conscientemente.
  • Esas decisiones causan harm material (financiero, psicológico, físico).

Ejemplos prohibidos:

  • Audio sublíminal que induces compras impulsivas.
  • Visual patterns que crean addictive behavior compulsive.
  • Personalization extrema que exploits vulnerabilities psicológicas (gambling addicts targeting).

Casos NO prohibidos:

  • Personalization standard de marketing (acceptable).
  • A/B testing para optimize UX (acceptable).
  • Algorithms que muestran content user is more likely to engage with (general recommendation, NO subliminal).

Línea: subliminal causing material harm. Both criteria needed para prohibition.

2. Explotación de vulnerabilidades

Sistemas que exploit vulnerabilities based on:

  • Edad (especialmente minors).
  • Disability.
  • Specific social or economic situation.

Para distorter behavior causando harm material o psicológico.

Ejemplos prohibidos:

  • Toy con AI que manipulates children to ask parents for purchases.
  • Sistema targeting elderly with cognitive decline para predatory loans.
  • Targeting adicts con personalized content que feeds addiction.

Crítico: el targeting de vulnerable groups + harm es la combination que activa prohibition.

3. Social Scoring (gobierno y empresa privada)

General-purpose social scoring by public authorities or private actors:

  • Evaluating o classifying personas based en social behavior, personality traits, or predicted characteristics.
  • Resulting en treatment detrimental o desfavorable in unrelated contexts.
  • O treatment disproportionate to the social behavior.

Ejemplos prohibidos:

  • China-style social credit system (gobierno).
  • Empresa rating customers based en general behavior across services para deny services.
  • Sistema "trustworthiness score" general-purpose.

Excepciones:

  • Sector-specific scoring (credit scoring solo para credit) - no unacceptable, but high-risk.
  • Legitimate business assessments within specific context.

Línea: scoring comprehensive across contexts y disproportionate treatment.

4. Predictive policing basado solo en profiling

Risk assessment of personas para predicting probabilidad de cometer crimen, basado solely en profiling (personality traits, etc.).

Excepciones:

  • Cuando se aplica para soportar human assessment (no predictive policing solo).
  • Basado en objective verifiable facts (no profiling).

Ejemplos prohibidos:

  • Algorithm predicting future criminality basado en demographics, neighborhood.
  • Sistema scoring "likelihood of recidivism" basado en profiling solo.

Excepciones legitimate:

  • Risk assessment basado en specific evidence y human review.
  • Forensic analysis para investigation specific.

Crítico: solo profiling es problemático. Combined con human review y objective evidence puede ser acceptable (high-risk, no unacceptable).

5. Untargeted facial recognition database scraping

Creando o expanding facial recognition databases by untargeted scraping of:

  • Facial images from internet.
  • CCTV footage.

Caso real prohibido: Clearview AI's business model — scraping social media para build facial recognition database de billions de personas sin consent.

Excepciones:

  • Targeted use con specific basis legal (e.g., investigating specific crime).

6. Emotion recognition en workplace y education

Inferencia de emociones of natural persons in:

  • Workplace.
  • Educational institutions.

Excepciones:

  • Medical use (emotion recognition para health).
  • Safety reasons (e.g., driver fatigue detection — debatable).

Casos prohibidos:

  • Sistema monitoring empleados emocional state durante trabajo.
  • Sistema evaluating student emotional engagement durante clase.

Razón: workplace y education son contexts donde power imbalance prevents free consent. Emotion recognition aquí es coercive.

7. Biometric categorization para sensitive attributes

Categorización biométrica to deduce:

  • Race, political opinions, religion.
  • Trade union membership.
  • Sex life or sexual orientation.

Excepciones:

  • Authorized labelling for datasets (with safeguards).
  • Law enforcement (specific limited cases).

Casos prohibidos:

  • Sistema clasificando personas por raza basado en biometric features.
  • Sistema deducing sexual orientation desde voice features.

Razón: estos atributos están protected especially under EU human rights framework. Biometric inference de ellos es violation de privacy y dignity.

8. Real-time biometric identification en espacios públicos

Real-time remote biometric identification by law enforcement en espacios públicos.

Excepciones (estrictas):

  • Targeted search por specific victims (kidnapping, missing children).
  • Prevention of specific imminent threats (terrorist attack imminent).
  • Localization or identification of suspects of specific serious crimes.

Requirements para excepciones:

  • Prior judicial authorization (with limited emergency exceptions).
  • Targeted (no mass surveillance).
  • Limited time.
  • Documentation.

Razón principal: mass surveillance via real-time biometric id es chilling effect en libertades fundamentales.

Cómo evitar accidentalmente caer en estas categorías

Even bona-fide use cases pueden activate prohibitions. Defense:

Defense 1: Document intent y context claramente

Si tu sistema es legitimate:

  • Document what specific purpose it serves.
  • Document who specifically will use it.
  • Document what safeguards prevent misuse.

Documentation matters cuando regulator pregunta.

Defense 2: Apply proportionality test

Para cada feature de tu sistema:

  • ¿Sirve un legitimate, specific purpose?
  • ¿Es proportionate al purpose?
  • ¿Están there alternatives que achieve purpose con less risk?

Si no podés justify each one con específicos, considerá removerlo.

Defense 3: Avoid feature combinations problemáticas

Combinations que activan prohibitions:

  • Vulnerable group targeting + harm potential → exploitation prohibition.
  • Comprehensive scoring + cross-context impact → social scoring prohibition.
  • Profiling + automated criminal prediction → predictive policing prohibition.
  • Biometric + sensitive attribute inference → biometric categorization prohibition.

Si tu sistema combines these, redesign.

Defense 4: Implement human-in-the-loop

Many borderline cases convert from "prohibited" to "high-risk" (still legal, just more obligations) when human decision making is preserved.

Example: predictive policing solo = prohibited. Predictive policing as input to human decision + objective evidence = high-risk (acceptable).

Defense 5: Geographical scope

Si tu sistema could activate prohibition pero solo deployás en non-EU markets, está OK desde EU AI Act perspective (otra regulation may apply en otros markets).

But: geofencing properly. EU users access = scope applies.

Qué hacer si tu sistema está en categoría prohibited

Honestidad incómoda: si analiza tu sistema y caés en unacceptable, opciones:

Opción 1: Re-architect

Pueden often eliminar el problematic component sin pérdida de core functionality:

  • Remove subliminal manipulation features → mantenés service.
  • Remove vulnerable group targeting → broadly applicable service.
  • Add human review to predictive policing → high-risk (acceptable).
  • Limit scope of biometric to sector-specific → high-risk.

Re-architect cuesta engineering time pero preserves business.

Opción 2: Geographic exclusion

If re-architecture is impossible, exclude EU users:

  • Geofencing.
  • Account creation restrictions.
  • Terms of service preventing EU use.

Trade-off: pierdes EU market. For some companies acceptable, for others existential.

Opción 3: Don't build it

Sometimes the answer is: this product shouldn't exist as designed. The market may not support it elsewhere either, given growing global regulatory alignment.

This is harder business decision. Sometimes correct.

Trampas comunes

1. "Mi sistema es 'inspired by' but no exactly..."

Regulators look at substance, no marketing. If function matches prohibited category, prohibition aplica regardless of how you describe it.

2. Underestimating "manipulation"

Many recommendation engines walk fine line. If you exploit cognitive biases para drive specific behavior, especialmente en vulnerable groups, consider exploitation prohibition.

3. "We have terms of service preventing this"

Terms of service ≠ technical safeguards. If users can misuse your system in prohibited ways, you may be liable regardless de what TOS says.

4. Treating "real-time" definition loosely

Real-time biometric identification incluye near-real-time. 1-minute delay is still "real-time" para regulatory purposes. Don't think you can dodge by adding small delay.

5. Asumiendo law enforcement excepción aplica broadly

Excepciones para law enforcement biometric identification son muy estrechas. Most use cases don't qualify. Default: assume prohibition unless explicitly within narrow exception.

Auto-verificación

1. ¿Cuál es la diferencia entre social scoring (prohibido) y credit scoring (high-risk)?

Social scoring (prohibido):

  • General purpose: evalúa comportamiento across multiple unrelated contexts.
  • Cross-context impact: scoring en context A causa treatment en context B (totally distinct).
  • Disproportionate: treatment es desproporcionado al behavior.

Example: government rates citizen based on shopping habits, social media, job performance, friends — and that score determines access to housing, transportation, employment.

Credit scoring (high-risk):

  • Sector-specific: only para credit decisions.
  • Single-context: financial behavior → financial decisions.
  • Proportionate: relevant data only (income, history, etc.).

Both involve scoring people. Differences: scope (general vs sector-specific) y proportionality (cross-context vs same-context).

Implication: legitimate sector-specific scoring (credit, insurance, employment) is high-risk (regulated, but allowed). Comprehensive cross-context scoring is prohibited.

2. ¿Cómo evitar que tu chatbot personalization caiga en "manipulation"?

Personalization standard NO es prohibido. La línea con manipulation:

Acceptable personalization:

  • Showing content user is more likely to engage with.
  • Recommendations based on stated preferences.
  • A/B testing UX para optimize satisfaction.
  • Adjusting tone basado en user style.

Potentially manipulative:

  • Exploiting cognitive biases (loss aversion, FOMO triggers).
  • Targeting psychological vulnerabilities (anxiety, addiction).
  • Subliminal techniques (hidden patterns affecting decisions).
  • Time-pressure or fear-based triggers que materially distort decisions.

Test: ¿user, if fully aware del mechanism, would still consent to it?

If "yes" (user understands the personalization helping them) → acceptable. If "no" (user would be uncomfortable knowing exactly cómo are being influenced) → potentially manipulative.

Documentation helps:

  • Disclose personalization features.
  • Avoid exploitation patterns.
  • Don't target vulnerable groups specifically.
  • Test with diverse users for unintended consequences.
3. ¿Por qué emotion recognition es prohibida en workplace pero no en consumer apps?

Razón principal: power imbalance.

In workplace:

  • Employee cannot freely refuse without job consequences.
  • Power asymmetry = no real consent.
  • Emotional state monitoring = coercive surveillance.
  • Could lead to discrimination based on emotional patterns.

In consumer apps:

  • User has choice to use or not.
  • Power balance more equal.
  • Consent can be more genuine.

Same logic in education:

  • Student cannot freely refuse without academic consequences.
  • Power imbalance prevents real consent.

Where emotion recognition might be acceptable:

  • Medical (with consent and clinical justification).
  • Voluntary consumer apps (mental health support, with clear consent).
  • Safety (driver alertness, with limited scope).

The principle: contexts where consent is meaningful = potentially acceptable. Contexts with power imbalance = prohibited or restricted.

Practical implication: HR analytics products that include emotion analysis must remove that feature or face EU prohibition. Substantial impact en HR-tech industry.

4. ¿Si mi sistema está en gray area, qué hacer?

Pasos prácticos:

  1. Document tu interpretation:

    • Describe el sistema.
    • Identify which category(ies) potentially apply.
    • Argue your interpretation con evidence.
    • Note alternative interpretations.

  2. Apply conservative test:

    • Default: assume more strict interpretation.
    • Comply con stricter requirements.
    • Less exposure si regulator interprets stricter way.

  3. Seek legal opinion:

    • For high-stakes systems, get formal legal review.
    • Documentation creates record of due diligence.

  4. Engage with regulator:

    • EU AI Office accepts informal queries.
    • Regulatory sandbox programs may be available.
    • Better to ask than guess wrong.

  5. Implement safeguards regardless:

    • Even if not technically required, implement strong safeguards.
    • Reduces real-world risk + future-proofs against regulation evolution.

  6. Re-evaluate periodically:

    • Regulator guidance will clarify gray areas over time.
    • Re-check your classification annually.

What NOT to do: assume "they won't notice" or "we'll deal with it if asked". GDPR enforcement showed regulators actively investigate. Same will happen for AI Act.

Resumen y siguiente paso

  • 8 categorías de unacceptable risk prohibidas absolutamente:

    1. Manipulación subliminal.
    2. Exploitation de vulnerabilities.
    3. Social scoring general.
    4. Predictive policing solo profiling.
    5. Untargeted facial recognition scraping.
    6. Emotion recognition en workplace/education.
    7. Biometric categorization para sensitive attributes.
    8. Real-time biometric ID en public spaces.

  • Bona-fide systems pueden accidentalmente caer: defense via documentation, proportionality, human-in-loop, geographic scope.

  • Si tu sistema está prohibited: re-architect, geographic exclusion, o don't build.

  • Penalties: hasta €35M o 7% revenue global.

Checkpoint: deberías poder identificar si tu sistema potentially activates cualquier prohibition.

Puente a la siguiente cápsula: la cápsula 03 cubre High-Risk — la categoría más extensa y la que más probablemente aplica a tu sistema. Vas a aprender exactamente cuáles use cases son high-risk, los criterios de qualification, y cómo identificar si tu sistema está allí.

Recursos

  1. EU AI Act Art. 5 — Prohibited Practices — texto.
  2. Future of Life Institute — Prohibited Practices Analysis — analysis.
  3. Clearview AI Cases — case study facial recognition.

Siguiente: 03-high-risk-categorias.md — High-Risk AI: categorías y criterios.

Cápsula 02 de 08 — Módulo 4 — AI Ethics & Compliance Guide

← Anterior1. Introducción al módulo: EU AI Act Deep DiveSiguiente →3. High-Risk AI: Categorías y Criterios