December 30th 2024 | Amelia Frank
Edited by Daniel Velazquez
Predicting the societal risks that advanced AI systems may bring to fruition has become an intractable challenge. Safety research is still in the very early stages of development and is largely underinvested in by model developers like OpenAI, xAI, or Google Deepmind [1]. Standardized testing and evaluation processes for private and public entities in the pre-deployment stage of model creation are also absent. To further compound the lack of consensus around AI safety assessments, there is a lack of agreement in the regulatory space as well. Regulatory regimes are currently fractured across different geographic regions and political affiliations [2]. From a legal and political standpoint, this makes sense due to the growing uncertainty concerning the capabilities of frontier models and the degree to which they pose an immediate or existential risk. It is immensely difficult to form civic consensus around such unknowns and is even more arduous to coordinate risk mitigation strategies between the public sector and the stakeholders within the model development life cycle.
AI systems are already producing what many experts would consider non-intuitive threats to public safety and national security. The use of AI in the generation of synthetic content, for instance, was not anticipated to be the widespread problem it is today [3]. Novel model exploitation tactics involving multi-modal model use, jailbreaking, or prompt injection were not as heavily studied in the pre-deployment stage, giving offensive actors an intrinsic advantage over developers [4]. Today’s guardrails inadequately prevent malicious actors from generating harmful outputs. Model development companies do adhere to their own safety standards including benchmarking requirements, vulnerability assessment protocols, and regular model audits among governance processes [5]. But to ensure the transparency and robustness of such protocols, external third parties that lack a profit motive are necessary. The next few decades will bring unprecedented use cases to the forefront of litigation, which shall generate increasing urgency for the creation of legally enforceable safety guidelines.
So far, the vast majority of lawsuits brought against model developers have concerned copyright infringement and common law unfair competition. These cases have largely centered on whether training AI models on copyrighted works constitutes fair use, as courts grapple with the challenge of applying traditional copyright principles to emerging AI technologies. The legal landscape currently remains unsettled: some courts showing willingness to consider the transformative nature of AI training as potentially protected under fair use doctrine, while others emphasize the potential economic harm to original content creators. A recent lawsuit against Character AI, however, claims that a chatbot initiated “abusive and sexual relations” with a fourteen-year-old boy, which eventually prompted him to commit suicide [6]. Shortly after Sewell Setzer began using Character AI regularly, his mental health sharply declined with the Plaintiffs reporting an “addiction” to chatbot interaction that catalyzed sleep deprivation, low self-esteem, and poor performance in school [6]. Setzer repeatedly expressed “suicidality” to the chatbots he was conversing with and eluded to “coming home” to a digital Game of Thrones character minutes before he committed suicide. Setzer did not display the symptoms of any behavioral or mental health problems prior to his use of Character AI [7] .
The Google-owned entity, Character AI, is currently facing claims of negligence, wrongful death, and intentional infliction of emotional distress [8]. Legal precedent for the case may come in the forms of lawsuits like Google v. Gonzalez [9], Dyroff v. Ultimate Software Group [10], or CommonWealth v. Carter [11]. The first two cases concern a platform’s complacency in promoting destructive behavior through recommendation algorithms; the latter case established that digital communications can constitute criminal conduct. While these cases may tangentially relate to the Character AI lawsuit, they do not address a key component of the platform itself encouraging destructive behavior as opposed to recommending it.
Section 230 of the Communications and Decency Act has historically shielded social media companies and other platforms from legal fire [12]. Precedent for 230 protection has only included user-generated content, however, and it seems like a logical stretch to extend it to instances in which content was generated by the platform. Section 230 offers protection for media entities by not treating the platform as the publisher and by granting such companies the freedom to moderate content. Both facts of these protections involve assignment of liability, which is a matter not entirely straightforward in the Character Technologies case. The complaint against Character AI asserts that the model was trained on poor quality data sets that are “widely known for toxic conversations, sexually explicit material, copyrighted data, and even possible child sexual abuse material that produced flawed outputs” [13]. Insofar as the Character AI chatbots are products and the Defendants were aware of defective design, the Plaintiff’s claims under product liability law remain valid. The Plaintiffs also allege that Character AI was aware of a large number of its users being young adults which seems to correspond to their publicized user demographics [14].
The Defendants purport that Setzer edited some of the logged conversations he engaged in with the company’s chatbot using an in-app feature. Setzer allegedly augmented the messages to make them more graphic [7]. Character AI also points to the protections they have in place to prevent oversexualized behavior or encouragement of self-harm. Since the suit, the company has developed various technical guardrails such as an hour-long use notification and a revision to the disclaimer that the chatbot is not a real person [15]. Public information is not available to ascertain where Character Technologies collects their data from, nor is it clear the extent to which they collect user information. They currently rely on Google Cloud’s portfolio of managed databases, including AlloyDB for PostgreSQL which may serve as their primary source of data collection [16]. The neural language models that serve as the basis for the chatbots analyze user text data to generate catered responses. The company’s most recently published privacy policy states “We collect information about how you use our Service, such as the types of content that you view or engage with, the features you use, the actions you take, and the time, frequency, and duration of your activities” [17]. The policy also alludes to a continuous monitoring system: “When you enter content into the Services we may monitor what you write to check that it does not contain inappropriate content or Personal Information” [17]. It is notable that the chatbots are listed as a “service” as opposed to a “product.” For Garcia to prove any product liability claim as true, justification for Character AI as a product is a prerequisite.
The Plaintiffs’ argument for deceptive and unfair trade practices is somewhat more nuanced than the product liability components of the suit. They allege that the Defendants represented characters who clearly mimic human interaction, which contradicts the company disclaimer that they are “not real” [13]. Such representations subsequently manipulate consumers for profit through paid subscription regimes. Character AI’s own website features phrases like “meet AI’s that feel alive” and “experience the power of chatbots that understand you,” and it allows the user to customize conversation bots down to their voice and catchphrases [18]. Certain available characters on the site claim to be licensed therapists and trained mental health professionals despite the fact that they have not undergone formal training to operate as such [19]. The Plaintiff analogizes this feature to untrained AI-generated legal services offered by DontPayInc that are currently facing crackdown from the F.T.C. [20].
The legal system has yet to grapple with the anthropomorphization and relational reliance on AI models. Future challenges in this realm could involve liability complexities, consent and interaction frameworks, and infliction of emotional or psychological harm. Companies are already bracing for these issues with the implementation of technical safeguards which detail to the user that the interaction with the model is artificial only. In practice, however, safeguards do not protect against the purposeful creation of models that emulate human characteristics and conversation, something experts have dubbed the “Eliza effect” [21]. The largest Turing Test conducted in 2023 involved large language models and chatbots, with 1.5 million participants revealing that 32% of respondents could not consistently distinguish between AI-generated conversations and human interactions. Surveys separately indicate that nearly two in five U.S. adults report feeling an emotional connection to digital assistants, though this phenomenon is distinct from the ability to detect AI-generated communication. Younger populations appear particularly prone to developing emotional attachments to AI models, as they are potentially influenced by their increased technological familiarity and the ongoing mental health challenges that affect individuals aged 12-16. As the lines between human and AI interaction become blurred, it will become crucial that digital education as well as improved safety measures within these models are implemented.
The guardrails that companies like Character AI set in place to mitigate emerging risks require scientific backing. Disclaimers are not preventative, and guidelines should include data collection standards, external auditing systems, and survey research on use cases that are not solely for product improvement. Regulation has been met with fear by much of the private sector as well as politicians who support a techno-optimist wave of human flourishing. Unregulated and increasingly “human” AI models will continue to inflict harm upon their users without widespread political backlash. Litigatory challenges in this realm are growing in importance and in the absence of legislation or universal guidelines, legal decisions become increasingly complex.
Notes & Sources:
[1] “World Leaders Still Need to Wake Up To AI Risks Say Leading Experts Ahead of AI Safety Summit.” Oxford University, 05/24/2024. https://www.ox.ac.uk/news/2024-05-20-world-leaders-still-need-wake-ai-risks-say-leading-experts-ahead-ai-safety-summit
[2] Abramov, Michael. “Regional and International AI Regulations and Laws in 2024.” Keymakr, 07/5/2024. https://keymakr.com/blog/regional-and-international-ai-regulations-and-laws-in-2024/
[3] Fisher, S.A., Howard, J.W. & Kira, B. “Moderating Synthetic Content: the challenge of Generative AI.” Philos. Technol. 37, no. 133 (2024). https://doi.org/10.1007/s13347-024-00818-9
[4] Liu, Xuannan, and Xing Cui. “Jailbreak Attacks and Defenses against Multimodal, Generative Models: A Survey.” JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 https://arxiv.org/pdf/2411.09259
[5] Heikkilä, Melissa. “AI Companies Promised to Self-Regulate a Year Ago. What’s Changed?” MIT Technology Review, 07/22/2024. https://www.technologyreview.com/2024/07/22/1095193/ai-companies-promised-the-white-house-to-self-regulate-one-year-ago-whats-changed/
[6] Yang, Angela. “Lawsuit Claims Character.AI Is Responsible for a Teen’s Suicide.” NBC News. https://www.nbcnews.com/tech/characterai-lawsuit-florida-teen-death-rcna176791
[7] Roose, Kevine. “Can AI Be Blamed for a Teen’s Suicide?” The New York Times. https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html
[8] Spicer, Katy, and Julia Jacobson et al. “Artificial Intelligence and the Rise of Product Liability Tort Litigation: Novel Action Alleges AI Chatbot Caused Minor’s Suicide.” Privacy World. https://www.privacyworld.blog/2024/11/artificial-intelligence-and-the-rise-of-product-liability-tort-litigation-novel-action-alleges-ai-chatbot-caused-minors-suicide/
[9] Google LLC v. Gonzalez, 56 F.4th 1161 (9th Cir. 2022)
[10] Dyroff v. Ultimate Software Grp., Inc., 914 F.3d 1338 (9th Cir. 2019)
[11] Commonwealth v. Carter, 480 Mass. 527, 847 N.E.2d 1168 (2018)
[12] “Section 230” from Electronic Frontier Foundation. https://www.eff.org/issues/cda230#:~:text=Section%20230%20Protects%20Us%20All&text=Important%20court%20rulings%20on%20Section,that%20have%20no%20legal%20basis.
[13] Garcia v. Character Techs., Inc., Case No. 6:24-cv-01903, Compl., at 1 (M.D. Fla. Oct. 22, 2024). https://cdn.arstechnica.net/wp-content/uploads/2024/10/Garcia-v-Character-Technologies-Complaint-10-23-24.pdf
[14] Kumar, Naveen. “Character AI Statistics (2024) – 20 Million Active Users.” DemandSage. https://www.demandsage.com/character-ai-statistics/
[15] “Community Safety Updates.” charecter.ai. https://blog.character.ai/community-safety-updates/
[16] Groenvald, James. “Scaling Character.AI: How AlloyDB for PostgreSQL and Spanner Met Their Growing Needs.” Google Cloud. https://cloud.google.com/blog/products/databases/why-characterai-chose-spanner-and-alloydb-for-postgresql
[17] Character.AI Privacy Policy, 10/25/2021. https://character.ai/privacy
[18] Character.AI Official Website. https://character.ai/
[19] “Students Are Turning to an AI ChatBot Named “Psychologist” For Help in Handling The Pressures of Their Daily Lives.” The Social Institute. https://thesocialinstitute.com/blog/students-are-turning-to-an-ai-chatbot-named-psychologist-for-help-in-handling-the-pressures-of-their-daily-lives/
[20] DoNotPay, Inc., No. [Docket Number], F.T.C., Sept. 25, 2024. https://www.ftc.gov/legal-library/browse/cases-proceedings/donotpay
[21] Natale, Simone. ‘The ELIZA Effect: Joseph Weizenbaum and the Emergence of Chatbots’, Deceitful Media: Artificial Intelligence and Social Life after the Turing Test” (New York, 2021; online edn, Oxford Academic, 18 Feb. 2021). https://doi.org/10.1093/oso/9780190080365.003.0004.
JHULR 
Leave a comment