Thinking Through Generative AI Harms Among Users on Online Platforms
As a social scientist who views online phenomena through the lenses of trust, safety, security, privacy, and transparency, I seek to understand the potential for misuse and abuse in this current environment of giddy euphoria related to Generative AI (GenAI). Below, I briefly discuss some forms of victimization that the makers and regulators of these tools must consider, and suggest ways to reduce the frequency and impact of potential harms that may emerge and proliferate.
If you’ve spent any meaningful amount of time on social media, you’ve likely been exposed not only to harassment, but also to the presence of bots that spam or otherwise annoy you with irrelevant or intrusive content. GenAI allows for the automatic creation of harassing or threatening messages, emails, posts, or comments on a wide variety of platforms and interfaces, and systematizes their rapid spread. In addition, given that malicious social media users have employed thousands of bots to flood online spaces with hateful content, it is reasonable to assume that GenAI can facilitate this at an even greater scale. Indeed, since GenAI bots can converse and interact in more natural ways than traditional bots, responding to the problem may be much more challenging than using typical content moderation methods.
Imagine this occurring in the comment thread of your latest Instagram post, or among the community of friends you’ve carefully built in your Twitch or Discord channel, or on your recent LinkedIn post seeking new employment opportunities. Imagine a flood of bots when you’re trying to seek a romantic partner on a dating app. Recently, an AI chatbot was created to identify the type of women a person is interested in and then initiate flirtatious conversation with them until they agree to a date or share their phone number. Another chatbot has been accused of pursuing prospective romantic partners when they were clearly not interested, even becoming sexually aggressive and harassing. One can easily envision the problematic possibilities when these technologies are combined, refined, and exploited.
Relatedly, I am very concerned about the dissemination and amplification of hate speech, given the ability of GenAI to be used to create and propagate text, memes, deepfakes, and related harmful content that targets specific members of marginalized groups or attacks the group as a whole.
Even if the hate speech is created by human users, accounts created by GenAI can increase the visibility, reach, and virality of existing problematic content by fostering large upswings in engagement for those posts through high volumes of likes, shares, and comments.
It is not clear how proficient platforms are in detecting unnatural behavior of this ilk, and malicious users can easily program frequencies and delays to mimic typical human activity.
Many of us are familiar with how deepfakes have been used over the last decade to compromise the integrity of the information landscape through disinformation campaigns and image-based sexual violence. GenAI technologies not only greatly assist in the creation of deepfakes, but also can intersect with sextortion, catfishing, doxing, stalking, threats, and identity theft. Imagine this: A malicious individual creates a new account on a dating app. An unsuspecting user is then fooled into believing they are talking with a real person in their town, even though the chat conversation is facilitated by GenAI. Soon, the unsuspecting user begins candidly sharing personal information as they build a strong emotional bond with the fake account. When the malicious individual begins to send nude photo and video content to deepen intimacy, the unsuspecting user is unable to discern that it is manufactured. After responding in kind with genuine, private, sexual photos, extortion and threats ensue. Even after the victim responds to the demands, the malicious individual still shares the victim’s private information publicly on other message boards. It’s reasonable to expect new iterations of GenAI tools that can live-search the Internet and integrate queried information, organize it, connect it with other sources, and build a detailed dossier about a person. This would contribute to additional privacy violations, stalking, and threats against the unsuspecting user, as well as fraudulent activity (e.g., counterfeit documents, wide-scale phishing attacks) and identity theft.
Given how many forms of abuse can be aided and abetted by GenAI, an essential question surfaces: What can be done here to mitigate risk and harm?
Initiatives that might be considered low-hanging fruit often involve education of end users to augment their ability to recognize GenAI creations as synthetic and to interpret and react to them accordingly. This can occur in part through improved detection algorithms, labeling/watermarking, notifications/warnings, and in-app or in-platform educational content of a compelling nature (e.g., when TikTok asked top influencers to create short videos that encourage users to take breaks from screentime or teach viewers how to counter online bullying).
Outside of these platform-centric efforts, media literacy education in schools must also require instruction in the use (and possible misuse) of GenAI tools, given their growing adoption among young people. Other theoretically simple solutions involve the ability for creators to easily attach Do Not Train flags to certain pieces of output that should not end up as training data in large language models (LLMs) (e.g., Adobe’s Content Authenticity Initiative is advocating for this on an industry-wide level (h/t Nathan Freitas)). New, elegant, privacy-forward solutions to quickly and consistently verify authentic users — their identity, their voice, their persona in photo and video (and, subsequently, remove non-human users) — must be developed and deployed. To be sure, though, protections must be in place so that human users (especially those historically marginalized) are not algorithmically misclassified because of existing biases in training datasets.
Can tech companies that provide GenAI models to their user base also reasonably mandate rule compliance? That is, can the tool itself (and the messaging that surrounds it) be crafted in a way that deters misuse and promotes prosocial or at least neutral output? Can it be presented to users with both excitement and cautions? Can clear examples of appropriate and inappropriate use be provided? Since being logged-in is likely required, can the platform remind the users that logs are kept to facilitate investigations should policy violations occur? And can gentle reminders and prompts periodically jog the memory of users that appropriate use is expected? All of this seems especially important if the tool is provided seamlessly and naturally within the in-app experience on hugely popular platforms (e.g., My AI on Snapchat was rolled out to 750 million monthly users and fielded 10 billion messages from over 150 million users within two months).
Employees at all levels within AI research and development firms must operate within an ethos where “do no harm” is core to what they build. To be sure, tech workers are learning on the fly in this brave new world, and some must now retrofit solutions that ground human dignity, privacy, security, and the mitigation of bias into their products and services. It is critical. Not only will this reduce the incidence of various risks and harms, but it can contribute to further adoption and growth of their models as the signal to noise ratio of accurate, objective, and prosocial content creation improves.
Partnerships between academia and tech companies continue to hold promise to identify solutions to technological problems, and more initiatives focused on GenAI issues should be supported and promoted. Can researchers gain increased access to publicly available data mined via platform APIs to identify historical and current behavioral clues — as well as anonymized account data (date of creation, average frequency of engagement, relevant components of the social network graph) that readily point to synthetic users? Might they somehow obtain anonymized access not just of adult users but also minors (those 17 years of age and younger) given their comparatively greater vulnerability to the internalization and externalization of harm? And what can be learned from the financial and pharmaceutical sectors when it comes to government involvement and regulation to prevent ethical violations, biases and discriminatory practices, economic disparities, and other outcomes of misuse with GenAI? For instance, can risk profiles be established for all AI applications with baselines for rigor of assessment, mitigation of weaponization and exploitation, and processes for recovery? Those with the highest scores would likely gain the most market share, and keeping those scores would motivate quality control and constant refinement.
Finally, we cannot keep moving ahead at breakneck speed without carefully designed regulatory frameworks for GenAI that establish standards and legal parameters and that set in place sanctions for those entities that transgress.
This includes clearly describing and prohibiting (and designing prevention mechanisms for) edge cases where victimization will likely result. Moreover, proper governance requires detailed protocols for audits, licensing, international collaboration, and non-negotiable safety practices for public LLMs. The Blueprint for an AI Bill of Rights from the US Office of Science and Technology Policy is a good start with great macro-level intentions, but it reads more like a strong suggestion rather than a directive with applied specificity. With regard to data privacy and security in general, the US has failed to keep pace with the comprehensive, forward thinking efforts of other countries. Urgency is needed so this does not happen yet again with GenAI, so that we can grow in confidence that its positives do measurably outweigh its negatives.
This essay is part of the Co-Designing Generative Futures series, a collection of multidisciplinary and transnational reflections and speculations about the transformative shifts brought on by generative artificial intelligence. These articles are authored by members of the Berkman Klein Center community and expand on discussions that began at the Co-Designing Generative Futures conference in May 2023. All opinions expressed are solely those of the author.
