Therapeutic AI: The New Regulatory Minefield

The AI Act is finally shedding its abstract skin and getting specific about 'therapeutic' and 'emotional support' AI. Recent guidance clarifies that systems designed for therapy, including general-purpose AI (GPAI) adapted for psychological support, are under the microscope. Providers now face a dual-layer compliance burden: you are responsible for the model’s foundational integrity and the specific deployment of your therapeutic application.

For us in the Belgian SME scene, this is a wake-up call. If your app uses an LLM to offer 'supportive chats' or mental health nudges, you aren't just selling software; you are potentially operating a high-risk medical-adjacent tool. The EU is moving away from the 'it's just a chatbot' defense. If your system influences a user's emotional state or mental health, you are expected to implement rigorous risk management systems, human oversight, and data governance that would make a traditional med-tech firm sweat.

Ignoring this isn't just a regulatory risk; it’s a liability nightmare. Belgian regulators are notoriously diligent when it comes to patient data and health-related services. If you are building in this space, you need to stop treating your AI as a black-box feature and start treating it as a clinical component.

This quarter, conduct a 'Regulatory Audit' on your product roadmap. Identify every feature that implies emotional support or therapeutic interaction. If you find any, map them against the AI Act’s high-risk requirements immediately. If you can’t document how your model handles a user experiencing a mental health crisis, you shouldn't be shipping that feature. It’s better to strip back a feature now than to pull a product from the App Store when the Data Protection Authority comes knocking.

Source: https://artificialintelligenceact.eu/ai-therapy-under-the-eu-ai-act/