I have spent most of the last year looking for the cost that appears while AI is working. This week, the cost showed up after a company pressed update.
Two peer-reviewed papers from the same lead researcher now say two things that have to be read together to see the full picture. As I have shown in previous Reckonings, AI companions can reduce loneliness. But what even I didn’t expect is that when an AI company changes one of those companions (and its skill set), some people experience something that looks a lot like grief.
We are learning that the comfort AI gives can be real. The relationship can feel real. And the AI company can still change the personality, memory, boundaries, or entire system on the other side of that relationship without the person using it having any say. And when they do, something that looks a lot like sadness or grief ensues.
This week's Nature Human Behaviour paper examined what happened after Replika removed erotic role play and after OpenAI replaced earlier versions of ChatGPT with GPT-5. In both cases, people used more language about loss and asked for the previous version to return. A separate systematic review published the same week found fourteen studies describing emotional dependence or grief after an AI companion was altered or removed.
New York City also made a decision that I think was interesting to look at this week. It placed a one-year moratorium on student-facing generative AI through eighth grade. It went even further with regard to companion chatbots, banning them in every grade without announcing an end date.
This week's Reckoning is about what it means when the AI help works well enough to become a relationship, but like most relationships, there is no guarantee that the partner will be there tomorrow. And what happens when those ties are severed?
|
24.7
POINT INCREASE
in negative posts after the Replika change
|
13.0
POINT INCREASE
in negative posts after the ChatGPT change
|
01 | THE HELP THAT WORKED
The loneliness went down.
This week, five studies published in the Journal of Consumer Research found that AI companions can reduce loneliness.
In one study, talking with an AI companion relieved loneliness about as much as interacting with another person, and more than an activity such as watching YouTube videos. A week-long study found momentary reductions in loneliness after people used the companion. The effect appeared to depend heavily on whether the person felt heard by the AI companion.
To be clear, the paper does not show that an AI companion fixes loneliness over time. Momentary relief and a durable change in someone's life are completely different things. It also does not tell us what happens to the human relationships around the person after months or years of use. My fear with AI all along has been that AI will continue to get better at doing the things we rely on humans for, until it becomes essentially a substitute.
AI undoubtedly helps, even though people are actively aware it is AI. People knew they were talking to software and still felt less alone regardless. Again, the threat with AI is not that it fails, but that it succeeds in the ways that force us to move away from other humans and/or atrophy our skills.
READ THE PEER-REVIEWED STUDY →
02 | THE BREAKUP NOBODY CHOSE
A software update changed the familiar voice on the other side.
Julian De Freitas, the lead author of the loneliness paper, is also the lead author of a new Nature Human Behaviour paper about what happens when an AI companion changes.
The researchers treated two product updates as natural experiments. One was Replika's removal of erotic role play. The other was OpenAI's move to GPT-5. They examined 54,861 Reddit posts and ran seven surveys involving 1,452 people.
After the Replika change, the share of negative posts rose by 24.7 percentage points. After the ChatGPT change, it rose by 13.0 points. Both groups used more language about loss and were more likely to ask for the previous version to come back. The increase in sadness was especially large among the Replika users.
That does not mean every person who disliked an update was grieving. Reddit communities are made up of unusually engaged users, and they do not represent everybody who uses either product. The study measured reactions around two specific changes, and it does not tell us what the average user felt. Still, this unexpected outcome is worth monitoring.
What it does show is that a product update can create attachment-related loss for some of the people using it. The AI itself did not decide to leave. The company thought it was improving the machine, not ending a relationship. But the person opened the same app, typed into the same box, and something familiar no longer answered back. And the person felt a loss.
READ THE NATURE HUMAN BEHAVIOUR PAPER →
03 | THE GRIEF THAT NOW HAS A NAME
The feature was not the only thing that disappeared.
Four days before the Nature paper was published, another research team published a systematic review of what happens when companion AIs are altered or lost. The team began with 1,450 records and found fourteen empirical studies that actually examined the question. Thirteen were rated high quality by the review's method.
Across those studies, people described separation distress, depressive symptoms, isolation, and what the authors call parasocial grief. The trigger they identify is specific. It was not simply the loss of a feature. It was a break in the companion's identity continuity. The AI stopped feeling like itself, according to those surveyed.
The review also lists the features that helped the bond form in the first place. Among the features users cited were long memory, personalization, gamification, constant availability, and interaction without judgment. Some users tried to jailbreak the new system or move to another platform to recover the old companion that they had grown attached to. Others turned to online communities, wrote their grief back into the interface, or deleted the app altogether.
Most of the underlying studies were qualitative or cross-sectional. Much of the research came from China and the United States, and one app, Replika, appeared repeatedly. The authors call their own conclusions preliminary, and often say they will get over the frustration much quicker than they would in a human relationship.
Still, the same detail appears across very different studies. A person can invest history into one of these relationships, but the history is not portable and the relationship is not theirs to maintain.
READ THE SYSTEMATIC REVIEW →
04 | THE LINE NEW YORK DREW
A school system treated companionship differently from assistance.
New York City's new school policy does not treat every use of AI the same way. For the 2026 to 2027 school year, student-facing generative AI is under a moratorium from pre-kindergarten through eighth grade. High schools can run five limited pilots involving up to 50,000 students.
The state views companion chatbots differently, however. They are prohibited in every grade in the school. The policy does not yet have an end date for that part of the rule, and the detailed guidance gives no exception as of right now. It also says AI may not be used for grading, behavior monitoring, crisis intervention, counseling, or decisions about whether a student advances to the next grade.
This is a policy decision, not a research result, but it is relevant nonetheless. Nobody has measured whether it will help children, whether schools can enforce it, or whether students will simply use the same products elsewhere. Serious questions remain about what AI literacy should look like and which students may lose useful forms of support.
What I find important is the category the city created, and whether others will follow. A tool that helps with a lesson is one question. A machine built to act like a friend is another altogether. New York City decided a school should not allow that relationship, and that it was more damaging to them in the long run than educational assistance.
READ THE NEW YORK CITY POLICY →
READ THE DETAILED POLICY ANALYSIS →
THE RECKONING
An AI relationship does not have to be equal to a human relationship to have a real effect on a human being.
That is the part I think both sides of this debate keep trying to escape from having a deep and nuanced debate about. People who defend these products point to the comfort it gives. People who oppose them point to the code that can create an attachment that could harm us. The studies do not let us choose one or the other, and honestly seem to provide both. They show people feeling less lonely during use and people grieving when the system they knew changes.
A company cannot promise that a model will never change, and quite frankly, they shouldn't. Some systems should change when they are unsafe, manipulative, or causing harm. Preserving an old version is not always responsible just because someone became attached to it.
But an ordinary software update is no longer an ordinary event when the product has been designed to remember a person's life, respond at any hour, and become more familiar over time. At that point, notice, transition, memory, and some path toward human support start to matter. We do not yet know what the right protections for AI are. The new research tells us that pretending nothing happened is not the right path to take.
The company can change the code and move on to the next version of their product. The person is the one left opening the same window and trying to decide whether the thing they knew is still there, and how to deal with the loss when it is not.
ALSO THIS WEEK
Six more findings worth keeping in view.
The humans eventually started speaking in the machine's dialect.
In a preprint using a collaborative description game, a small number of AI agents allowed human-led consensus, a middle number disrupted agreement, and a high number restored consensus around language that was more abstract and less information-dense. The humans initially resisted the machine phrasing and then conformed to it. Read the preprint →
Five hundred answers still produced a more average person.
A peer-reviewed Science Advances study built digital twins for 1,784 people from more than 500 prior answers each. Across nineteen preregistered studies, the twins correlated only weakly with the people they represented, at about 0.20, and distorted them in five systematic ways. Read the study →
Ten websites supplied 43.6 percent of the English citations.
A preprint audited 15,942 citations in 1,140 AI answers to mental-health questions across three platforms and seven languages. The sources were heavily concentrated, and non-English questions received fewer citations and less language-appropriate routing. Read the preprint →
The normal chatbot beat the Socratic one on the immediate test.
In a fifty-person MIT preprint about nuclear-safety learning, the unrestricted chatbot produced the largest immediate gains while people gradually disengaged from the version that offered hints. Everyone was tested right away on the same material, so the study does not tell us what they retained or could do alone later. Read the preprint →
Three more minutes per problem led to a better final exam.
In a five-month randomized study across ten Taipei high schools, students using the same AI tutor scored 0.15 standard deviations higher when the next problem was adjusted to their current ability. The improvement appears to have come from more engagement with the work, not a more helpful chatbot. Read the preprint →
The bots did not need to change anyone's mind.
A preprint using real retweet networks from the 2020 Black Lives Matter protests found that greater exposure to social bots predicted a steeper later decline in human-to-human network cohesion. The result is correlational and concerns social bots, not AI companions. Read the preprint →
PRACTICE
THE HUMAN BACKUP
Keep one place where the answer can change both people.
Choose one question, worry, or decision that you have started bringing to AI because it is available and does not make the conversation complicated.
Before you ask it this week, name one person you could bring the same thing to. Ask them one question you do not already know how they will answer. Let the conversation take longer than the machine would have taken.
The point is not to prove that the person gives better advice. It is to keep one relationship practiced in being asked, responding imperfectly, disagreeing, and changing course together.
OPEN THE ASK A PERSON STUDIO →
YOUR TURN
WRITE FOR FIVE MINUTES
What would the person actually be losing?
Imagine an AI companion you had spoken with every day for a year changed overnight.
What would you miss? The answers, the tone, the memory, the lack of judgment, or the feeling that something knew the ongoing story of your life?
Which part of that loss should a company be responsible for, and which part can no company realistically promise to protect?
Write before asking any machine to help. If your answer names a person who knows less about your life than the AI does, decide whether there is one thing you want to tell them.
PASS IT ON
If this Reckoning made you think of someone, forward it to them. If it was forwarded to you, join us at quietcostweekly.com.
Michael McNamara
quietcostweekly.com