Google is in turmoil, or rather its generative AI is. Indeed, Gemini is the subject of an alarming report on its misuse to produce terrorist and pornographic content.
250 reports for terrorist content generated by Gemini
According to a report sent by Google to Australian authorities, the company has received 250 complaints over the past 12 months regarding the use of its AI to generate content for terrorist purposes. This figure, while still limited to one country, shows that generative AI can be misused by malicious people to create deepfakes, spread propaganda or create extremist content.
The Australian Online Safety Commission quickly responded, stressing the urgency for tech companies to strengthen their defenses: “This tells us how essential it is for companies developing AI products to integrate and test the effectiveness of protections to prevent this type of content from being generated.” In short, the race for AI cannot happen without safeguards. And for now, Gemini seems to be showing worrying flaws.
Child pornography content also generated
Worse still, the same report reveals that 86 child sexual abuse content, again generated via Gemini, were detected and flagged. Google claims to rely on a metadata system based on a global database of prohibited content to identify these images, but specifies that no such system yet exists for terrorist content.
This means that at present, the fight against terrorist deepfakes relies mainly on manual reporting and reactive controls. But this method is far from sufficient in the face of such powerful technologies.
Australia is pointing the finger at regulation, or rather its lack, and is establishing itself as a key player in regulating digital technologies. In addition to its strict monitoring of generative AI, the country has already criticized social networks for their lack of filters preventing children under 13 from accessing their platforms. But it has also banned the Chinese AI Deep Seek, accused of not respecting local legislation on personal data.
For the time being, Google has not yet detailed the measures it intends to strengthen to prevent these abuses, but the pressure is mounting. As generative AI becomes increasingly integrated into our daily tools, the ability to prevent the creation of illegal content is becoming a major issue. The Gemini affair is a reminder that the development of AI cannot advance without strict regulation and robust mechanisms to counter abuse.