San Francisco Orders Apple and Google to Remove AI 'Nudify' Apps
San Francisco city attorney David Chiu has sent cease-and-desist letters to Apple and Google, demanding the removal of 13 apps from their app stores that allow users to create AI-generated nonconsensual nude images. The letters, seen by WIRED, accuse the tech giants of profiting from and enabling the distribution of harmful deepfake technology.
Legal Action and Demands
On Thursday, Chiu's office sent legal notices to both companies, stating that the apps violate California laws prohibiting the creation and distribution of deepfake pornography. The apps, marketed as face-swapping tools, have been found to facilitate the generation of explicit images without consent. Chiu emphasized that such technology is illegal, harmful, and unacceptable, and called for stricter moderation to prevent these apps from appearing in app stores.
"Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu told WIRED. He added that Apple and Google have likely profited from these apps through in-app payments, and they must take responsibility for ensuring that their platforms do not facilitate sexual abuse.
Companies' Responses
Google spokesperson Dan Jackson confirmed that the company has removed hundreds of apps with nudifying features, including five flagged by Chiu's office. Google claims to have taken proactive steps to restrict access to such apps and has suspended search terms like 'nudify' on its platform.
Apple spokesperson Adam Dema stated that the company has removed three of the flagged apps and is terminating the developers' accounts. Apple also noted that users can report problematic apps, and the company has strict policies against apps that generate or distribute pornographic content.
The Scale of the Problem
Researchers have repeatedly identified apps on Apple's App Store and Google's Play Store that enable the creation of sexual deepfakes. These apps often require minimal effort to produce realistic images or videos, with some services offering results for free or at a small cost. The harmful impact of such technology has been well-documented, with victims facing bullying, humiliation, and mental health issues.
The 13 apps targeted by Chiu's office, eight on the App Store and five on the Play Store, advertise themselves as face-swapping tools but include features that allow users to create sexual deepfakes. One app, with over 1 million downloads, promotes AI-generated images in styles like 'bikini queen curvy' and 'cinematic intimacy.'
Ongoing Challenges
Despite policies prohibiting pornography and abuse, both Apple and Google have struggled to completely eradicate such apps from their platforms. Previous reports by watchdog groups and researchers have identified hundreds of problematic apps, some of which have been downloaded millions of times and generated significant revenue.
Chiu expressed his commitment to pursuing legal action if necessary, stating, "It's our hope that these companies will do the right thing—but if they don’t, we will have to consider all of our legal options."
Conclusion
The demand for the removal of AI 'nudify' apps highlights the urgent need for stricter regulation and enforcement to combat the misuse of deepfake technology. As the technology advances, the responsibility falls on tech giants like Apple and Google to ensure their platforms do not enable harm.