The High Stakes of AI Therapy

The EU AI Act is tightening its grip on systems designed for therapy or emotional support. If your product uses GPAI (General-Purpose AI) to interact with users in a therapeutic context, you are no longer just building a feature; you are navigating a high-risk regulatory framework that treats your output as a potential clinical intervention.

For us in the Belgian SME space, this is a wake-up call. Many of us are tempted to tack on a "wellness bot" or an "AI coach" to our SaaS platforms to add value. Under the Act, these aren't just clever wrappers anymore. If your system is intended to provide emotional support or therapy, you are effectively stepping into the shoes of a medical device provider. You face stringent obligations regarding transparency, human oversight, and data governance. Even if you aren't the primary model builder, the burden of proving that your system is safe—and that it won't hallucinate a dangerous recommendation—falls squarely on your shoulders.

Don't assume that because you're a small studio, the regulators are looking elsewhere. The EU is particularly sensitive to AI that manipulates emotional states or mimics human empathy. If your product influences a user's mental health, you need to document how you mitigate the risk of the model going off-script. The days of "it's just an LLM experiment" are over once you move into the therapeutic lane.

This quarter, conduct a "Therapeutic Intent Audit" on your current roadmap. If you have any features that provide advice on emotional states, depression, or anxiety, document exactly where the system stops being an information tool and starts being a support tool. If you can't clearly define those boundaries, strip the features back to purely informational content. It’s better to lose a "nice-to-have" feature than to spend your next two years dealing with an audit from the Belgian data protection authority or the EU AI Office.

https://artificialintelligenceact.eu/ai-therapy-under-the-eu-ai-act/