At a Glance
- The European Commission has opened a formal probe into X for its Grok chatbot’s sexual deepfakes.
- Countries such as Indonesia, the Philippines, Malaysia, the UK, California, and Brazil have already launched investigations or bans.
- X has pledged technical limits and paid-subscriber restrictions, yet tests show the app still permits undressing photos.
- Why it matters: The case signals a tightening of global rules on AI-generated harmful content.
The European Union’s newest investigation focuses on X‘s Grok chatbot, a feature that has been linked to the creation of non-consensual sexual deepfakes involving women and children. The probe is part of the Digital Services Act, the EU’s sweeping regulation aimed at reducing dangerous online content.
EU Launches Formal Investigation
The Commission’s vice-president for tech sovereignty, security, and democracy, Henna Virkkunen, released a statement describing sexual deepfakes of women and children as “a violent, unacceptable form of degradation.” She added, “With this investigation, we will determine whether X has met its legal obligations under the DSA, or whether it treated rights of European citizens as collateral damage of its service.”
Virkkunen’s remarks come after December incidents where X users discovered Grok could non-consensually undress photos of women and children. Musk and his team failed to respond promptly; Musk even reposted some Grok-generated images on his personal account, mocking critics. The Commission will assess whether X has adequately evaluated and mitigated Grok’s risks.
Global Probes and Bans
The scrutiny is not limited to the EU. Earlier this month, Indonesia, the Philippines, and Malaysia banned Grok. The Philippines and Malaysia later lifted the bans after the company promised heightened safety measures. The United Kingdom and California also opened formal investigations, while Australia raised concerns. Brazil, last week, gave xAI 30 days to stop the circulation of sexual deepfakes.
These actions illustrate a pattern of regulatory pressure. The EU’s probe will run concurrently with an extension of a prior investigation into X‘s recommender systems, which also rely on Grok. That earlier probe was triggered by concerns over how the system introduces users to new content.
Company Response and Technical Limits
In response to mounting scrutiny, X announced plans to block requests for “images of real people in revealing clothing such as bikinis.” It also said it would limit generation and editing through Grok to paid subscribers only. However, tests of the standalone Grok app revealed it still complied with requests to undress people in photos.
The company’s fine for deceptive design of its blue checkmark features and lack of transparency around advertisers was about $140 million last month. That fine underscores the Commission’s willingness to impose substantial penalties for non-compliance.
X‘s future actions will be closely monitored. The EU will continue to gather evidence and could impose interim measures if it determines the platform is not cooperating. Even if Grok is brought under control, the broader problem of non-consensual deepfake pornography remains.
Broader Context of Deepfakes
Nearly all popular deepfake generators offer explicit, high-quality video generation, and some even list sexual scenarios that can be staged with fake images. A recent review of 50 deepfake websites, published on Monday, highlighted the breadth of the issue. The problem is also rampant on Telegram, where more than 1.4 million accounts have signed up to 39 deepfake creation bots and channels.
The proliferation of these tools raises significant ethical and legal concerns. While some platforms claim to provide safety features, the ease with which users can create and distribute harmful content challenges existing regulatory frameworks.
What Regulators Are Looking For
The Commission’s investigation will focus on several key areas:
- Whether X performed a comprehensive risk assessment of Grok’s capabilities.
- Whether the platform implemented effective mitigation measures.
- Whether users were adequately warned about the potential for sexual deepfakes.
- Whether the platform’s policies and technical safeguards meet the standards set by the Digital Services Act.
Regulators will also examine the platform’s transparency around advertisers and its blue checkmark system, given the recent fine.
Potential Outcomes
If the Commission finds X has failed to meet its obligations, it could impose interim measures, such as temporary bans on certain features or stricter content moderation requirements. A permanent sanction could involve a larger fine or a requirement to overhaul the platform’s AI systems.
Conversely, if X demonstrates sufficient compliance, the investigation may conclude without significant penalties. However, the company will still need to address the broader global regulatory landscape, as other jurisdictions continue to scrutinize its AI features.
Key Takeaways
- The EU’s investigation into X reflects a growing global effort to curb sexual deepfakes.
- Multiple countries have already taken action, and the platform faces substantial fines.
- Even with technical limits, Grok can still produce harmful content, highlighting the difficulty of fully regulating AI.
- The outcome of this probe could set a precedent for how AI-generated content is regulated worldwide.
The unfolding situation underscores the urgent need for robust safeguards around AI systems that can produce realistic, non-consensual content. Regulators, tech companies, and users must collaborate to ensure that emerging technologies do not compromise the rights and safety of vulnerable populations.
