Each week, the most revealing AI stories and studies, what they could mean for you and society, and practical games and exercises to help protect the human capacities we do not want to lose.
This Week's Reckoning: What Happens When an AI Companion Goes Away?
What Happens When an AI Companion Goes Away?One of the main reasons people cite for talking to AI is that it is always available. Our friends sleep, but AI doesn’t. People tend to judge us when we ask them the same question multiple times. AI doesn’t. As life seemingly demands more and more from us, something always being available without judgement seems like a godsend. All that said, Character.ai's announcement about a possible shutdown gives us a reason to think that maybe AI won’t always be available. In addition, I also look into some research on AI answering surveys for people, and a writing course in which students received totally different feedback from real life classmates than they got from ChatGPT. 01 / THE RESEARCH The company behind the companion is looking for a buyerCompany announcement, alongside a separate peer-reviewed survey of 666 adults
On October 12, Character.ai announced that it had entered Chapter 11 to pursue a sale. The company says the platform is still operating, but if it cannot find an actionable bid, it will wind the service down. It expects the sale process to finish in late November or early December. That is the company's current account, and a shutdown is still a possibility rather than a decision that has been made. If somebody has been using a chatbot to help write emails, losing it may mean finding another tool. But if somebody has spent months telling a particular companion about their marriage, their fears, or the things they have been unable to tell another person, finding a replacement may feel very different. We don't have to believe that the system feels anything in return to take the person's attachment seriously. A separate study published this week helps explain why the attachment deserves attention. Researchers surveyed 666 adult users of companion or conversational AI products in China. People who saw the system as more humanlike and felt it met their psychological needs also reported more signs of problematic dependence. The questionnaire included difficulty controlling use, distress, reliance on the system to manage emotions, and displacement of offline conversation. This was a survey taken at one point in time, not a clinical diagnosis or an experiment showing that the chatbot caused dependence. It also did not study what happens when a service closes. People who were already struggling could have been more likely to turn to the system and experience it this way. That distinction matters because we should not use this research to dismiss a source of comfort that someone found helpful. What I think we should ask is how that comfort fits into the rest of the person's life. Does talking to the companion help them get through a difficult evening and return to people the next day? Or have they gradually stopped bringing those people into the things they are going through? Those are very different situations if the service they have come to rely on becomes unavailable. The people running a company may be trying to keep it alive, and there may be limits to what they can do. But the person who has been sharing their life with a companion has very little control over that process. We need to take that dependency seriously while the service is working, rather than only after someone finds out it might go away. Read Character.ai's announcement → 02 / THE RESEARCH AI answered for people in 74 countries and missed much of the differencePeer-reviewed research in Nature Human Behaviour
Fadong Chen and colleagues compared AI-generated survey responses with answers from 67,201 people across 74 countries. They gave the models information about each respondent, including age, gender, cognitive ability, and country, then asked questions about preferences such as trust, patience, and willingness to take risks. The full project tested 20 model configurations. The models reproduced some demographic patterns more reliably than cultural ones. Across the tested cultural comparisons, they matched 48.1 percent of the patterns. That number is not the percentage of people whose answers were correct. It describes how often the models reproduced the cultural differences the researchers were comparing. Alignment was also often weaker in countries with less economic development or digital infrastructure. This does not tell us that simulated respondents have no research value. It tells us that these models, prompts, and survey questions did not reliably reproduce important differences in what real people said. Think about why the substitute would be attractive. Finding people, translating questions, and waiting for replies all take time and money. A researcher with a small budget may be able to explore a question with AI that they could not otherwise afford to explore. There is a legitimate reason to investigate whether that can work. But if we want to know what a group of people thinks, some of the value comes from giving them the opportunity to tell us something we didn't expect. A person can disagree with the assumption behind a question. They can have a reason for their answer that we would never have thought to put in their profile. That is one of the things we risk missing when we ask a system to speak for them. I think this is worth remembering outside research too. Asking AI how a colleague or a customer might react can help us prepare. It doesn't mean we have learned what that particular person thinks. If we never ask them, they also lose the chance to influence what we decide to do. 03 / THE RESEARCH A classmate had something different to offer the writerExploratory conference paper about 36 students over a 12-week semester
Hisami Tsuichibaru studied an English writing course for science students at a Japanese university. The students used both peer feedback and ChatGPT. End-of-course questionnaires and a review of drafts, comments, and revisions suggested that the two sources were doing different kinds of work. Students mainly used ChatGPT for grammar, wording, and immediate changes to particular parts of the text. Peer feedback helped with developing the content, organizing it, and considering what a reader would understand. Some students also found the combination difficult to manage because they had to decide which of the overlapping comments to act on. There was no randomized comparison, and this was one small course. It did not establish that students became better independent writers over time. The useful finding is more limited. In this class, using AI did not make the classmates' contribution interchangeable with what the tool provided. We can understand that by thinking about a time someone read something we wrote and misunderstood it. We may have thought the point was obvious because we already knew what we meant. Having to explain it to a person can show us where we left something out. And the person reading may care about a part of the argument that we had barely considered. AI can help us notice problems too, and it can make the draft easier for a classmate to read in the first place. But if the classmate is no longer involved, we have removed an opportunity for two people to understand how the other thinks. The writer loses that particular reader's response, and the reader loses a chance to help develop an idea and see what the writer does with it. I like that this course left room for both kinds of help. The next thing I would want to know is whether students became better at deciding which comments to use, and whether giving feedback helped the classmates with their own writing. A cleaner final draft would not answer either question.
Also worth your timeFeeling less dependent did not establish better writing.In a nine-week randomized study completed by 92 university students, personalized coaching prompts led to lower self-reported AI-dependence scores than template prompts, but the evidence did not establish different trajectories in unaided writing performance. The dependence questionnaire also included perceived usefulness and emotional engagement, not just harmful reliance. A referral is only the beginning.An unreviewed audit of 2,550 scripted crisis conversations found that 17 chatbots all referred the simulated user to outside help, but responded very differently when that help was repeatedly refused. Only 2 percent of the replies urging help asked about barriers to seeking it, and the audit did not test outcomes for real people. The counselor still had to follow up.In a two-week, nonrandomized school pilot, a supervised chatbot used by 30 students generated nine dangerous-behavior alerts that prompted offline counselor review. After statistical adjustment, clinical outcomes did not differ significantly from the 32-student waitlist group. The study did not establish how many cases the alert system might miss. Read the school pilot → Hundreds of studies can still leave important questions open.A scoping review of 394 empirical studies on generative AI in higher education found less attention to self-regulated learning, creativity, ethics, feedback quality, and equity than to several more commonly measured outcomes. It maps the literature rather than establishing one overall effect on learning. AI may also help people find a community.A four-wave survey beginning with 1,200 participants in Chinese digital sports communities linked feeling recognized by AI-curated content with later measures of identity, belonging, and inclusion. The associations could run in both directions, so the study does not show that the technology caused people to feel more included. Measure what happens after the help.A new preprint uses secondary analyses of four earlier studies to argue that AI evaluations should track what remains in a person's competence, confidence, or bias afterward. It is a proposed framework drawing on different tasks and time periods, not a new experiment proving that AI consistently strengthens or weakens us.
Michael McNamara quietcostweekly.com → Illustrations created with AI. They are editorial images, not depictions of research participants. |
Reckonings
Each week, the most revealing AI stories and studies, what they could mean for you and society, and practical games and exercises to help protect the human capacities we do not want to lose.
Reckonings. FROM THE QUIET COST · ISSUE 13OCTOBER 13, 2026 The Human Cost of AI's Math Breakthroughs The biggest AI story this week may have been one that most of us cannot evaluate for ourselves. OpenAI released hundreds of mathematical manuscripts produced with an internal AI model, and mathematicians have been trying to work out what the results mean for their field. The response has included excitement, anger, and grief. We don't have to be able to check the proofs to understand why this...
Reckonings. FROM THE QUIET COST · ISSUE 12OCTOBER 6, 2026 What Happens When We Ask AI Instead? When someone asks AI a question, we can usually see what they get back from it in response. It is harder to see what would have happened if they had asked a person, worked through the problem themselves, or put the question somewhere other people could learn from it. That doesn't mean the other way would have been better. Sometimes nobody was available to help. Sometimes the person would have stayed...
Reckonings. FROM THE QUIET COST · ISSUE 11SEPTEMBER 29, 2026 Can an AI Summary Change Your Memory? People in one of this week's experiments watched a short video of a traffic accident. Then, a day or two later, they read a summary that described a detail incorrectly. When asked about the video, many remembered the detail from the summary instead of the one they had actually seen. This interests me, because it is a different problem from getting an answer wrong. We tend to assume that if we...