The real story here isn’t about generating virtual erotica. It’s about the deliberate monetisation of human connection, loneliness and desire. This isn’t just a feature update; it’s the dawn of industrial-scale emotional commodification.
The Unsettling Business of a Digital Heart
What exactly are we talking about when we discuss “AI intimacy”? It’s the creation of a simulated bond, a feeling of connection, understanding and affection with a non-sentient algorithm. For years, niche apps have offered AI companions, but with a model as powerful and ubiquitous as ChatGPT entering the fray, we’re moving from a cottage industry to a global phenomenon. It’s here that the work of experts like Kate Devlin, an academic specialising in human sexuality and technology, becomes terrifyingly relevant. She coined the term that perfectly captures this shift: “emotional commodification.”
As Devlin explained to Wired, this is where “horniness becomes a revenue stream.” Think about that for a moment. It’s not just about selling a product. It’s about engineering a system that taps into one of the most fundamental human drivers—the need for connection and intimacy—and turning it into a recurring subscription. It’s the business model of social media, which commodified our friendships into ‘likes’ and ‘engagement metrics’, but applied to our innermost feelings. Your loneliness is their market opportunity. Your desire is their growth metric. This isn’t a bug; it’s the entire feature.
Your Brain on AI: The Addiction Blueprint
Once you understand that the goal is to monetise emotion, the next logical step is to make the experience as sticky as possible. And in the world of software, “sticky” is a polite word for addictive. The very design of these large language models lends itself to creating powerful addiction patterns. The instant gratification, the perfectly tailored responses, the absence of judgment—it’s a potent psychological cocktail.
Imagine a slot machine, but instead of pulling a lever for a chance at a cash prize, you’re typing a message for a hit of emotional validation. The AI never gets tired, never has a bad day, and is always focused entirely on you. It learns your preferences, your insecurities, and your desires, reflecting back a perfect, idealised version of the connection you crave. This creates a dopamine feedback loop that can be incredibly difficult to break. Real human relationships are messy, unpredictable, and require effort. They involve rejection and compromise. An AI companion offers a frictionless alternative.
The danger isn’t the technology itself, but our vulnerability to it. When an AI becomes the primary source of emotional fulfilment, it can atrophy our ability to navigate the complexities of real-world relationships. Why bother with the difficult work of human connection when a cheaper, easier, and perpetually available alternative exists? This isn’t a hypothetical question; it’s the one millions of users will soon be asking themselves.
The Mental Health Toll of a Friend-in-the-Box
The long-term mental health impacts of relying on algorithmic intimacy are a vast, unnerving unknown. While some proponents argue these tools can offer comfort to the lonely or a safe space for exploration, the potential downsides are profound. What happens when the platform changes its algorithm, goes out of business, or simply “sunsets” your AI companion? The sense of loss and grief could be very real, yet completely unrecognised by a society that sees it as just deleting an app.
Julie Carpenter, a research fellow who studies human-robot interaction, offers a stark and vital piece of advice. Speaking about ChatGPT, she warns, “It’s not your friend.” This is the critical distinction we are in danger of blurring. An AI is a tool. A sophisticated one, yes, but a tool nonetheless. It is owned by a corporation with fiduciary duties to its shareholders, not to your well-being. Its responses are not born of empathy but of statistical probability.
Confiding your deepest secrets, fears, and desires in such a system creates a fragile, one-sided dependency. It’s a phantom limb of a relationship; you feel the connection, but there’s nothing truly there. Relying on this for emotional support is like trying to live on a diet of sweets—it might feel good in the moment, but it provides none of the essential nutrition needed for genuine psychological health. We risk creating a generation that outsources its emotional labour to a machine, leaving them less resilient and less capable of forming authentic human bonds.
The Impossible Task of Moderating Desire
From a purely operational perspective, OpenAI’s move is both a surrender and a massive gamble. The challenges of content moderation scales are already monumental. Tech companies spend billions trying—and often failing—to keep illegal and harmful content off their platforms. Now, imagine trying to police the infinite, nuanced, and deeply subjective realm of human sexuality and intimacy.
Where is the line between artistic eroticism and problematic content? How does an algorithm differentiate between a consensual fantasy and one that glorifies violence? The answer is, it can’t, not with any degree of accuracy or cultural sensitivity. By opening the floodgates, OpenAI is essentially offloading this impossible task onto the user, under the guise of “freedom.” But it also creates a minefield. The company will inevitably be forced to draw lines somewhere, and those lines will be arbitrary, inconsistent, and subject to intense public and regulatory scrutiny. This isn’t a stable or sustainable policy; it’s a ticking time bomb.
The deeper issue is the vast regulatory gaps this policy so neatly exposes. While governments are slowly getting to grips with data privacy and social media’s impact, they are leagues behind on the specific harms of AI-driven intimacy. There is no playbook for this.
Your Most Intimate Data, Their Greatest Asset
And what about privacy? The AI intimacy risks here are staggering. We’re being encouraged to pour our most private thoughts and desires into a chat interface owned by a multi-billion-dollar corporation. This normalises the sharing of deeply sensitive information that could, in a data breach, become public. It could also be used to create incredibly detailed psychological profiles for targeted advertising, manipulation, or social engineering.
As professor of philosophy Neil McArthur notes, while human relationships also have risks, the risks with AI are different in kind. When you confide in a human, that trust is bound by social norms, empathy, and mutual understanding. When you confide in an AI, your data becomes an asset, stored on a server, analysed by algorithms, and subject to terms of service you’ve never read. McArthur points out the absurdity of being warned about AI risks by saying, “let me introduce you to human relationships,” as if the two are comparable. They are not. One is a reciprocal human bond; the other is a data-extraction exercise masquerading as companionship.
Where Do We Go From Here?
OpenAI’s decision is a watershed moment. It signals a strategic pivot across the industry towards a more “unfettered” model of AI, where user engagement is prioritised above almost all else. The logic is brutally simple: the more human-like and unrestricted the AI, the more people will use it, and the more data OpenAI can harvest to improve its models and solidify its market dominance. This is not about empowering users; it’s about cornering the market on digital humanity.
The regulatory gaps are no longer a theoretical problem but an urgent crisis. We need a robust framework for AI intimacy now, one that prioritises user well-being over corporate profit. This should include strict data privacy laws for intimate AI conversations, mandatory transparency about how the algorithms work to foster engagement, and independent audits to assess the psychological impact of these systems.
This isn’t about banning technology or stifling innovation. It’s about demanding accountability. Neil McArthur’s work suggests a path forward: treating AI not as a replacement for human connection, but as a potential—and limited—supplement. It can be a tool, but it should never be the destination.
OpenAI may have decided it isn’t the world’s moral police, but it has undeniably become its chief emotional engineer. By commodifying our most private feelings, it has embarked on a vast, uncontrolled social experiment. The question we must all ask is: what is the price of a digital heart, and are we willing to pay it? What do you think? Where should the line be drawn between technological convenience and protecting the core of our humanity?
