| Reckonings |
ISSUE 17 NOVEMBER 10, 2026 |
FROM THE QUIET COST
Can AI Get to Know Us and Still Get Us Wrong?
When we ask someone for help, it is not always just the actual help with that project that we are looking for. Sometimes we have had a bad day and just want to have somebody to talk about it with, while we keep our hands busy. Other times, we really do need help with a specific thing, and we are looking for them to show us how to do it. Sometimes, we want to hear about or see where we fell short, and other times, it’s the last thing we want to hear about when we are getting that help. And it can be the same person we go to in both cases, because we trust that they know us well enough to understand when we need one thing, and not the other.
The studies this week made me think about a world where we go to AI for help, more than we go to people, and whether AI will be able to tell the difference and adjust accordingly. One study looked at how an AI’s memory of someone who asked it to be kind changed the feedback it gave that person on unrelated work later. Another study gave writers an AI tool to see if it would help or hurt their ability to write more clearly, and closer to the original goals that they had when they set out to write the post. And in a third study, engineering students used AI to check their thinking, or their process, but could not use it to ask for the answer itself. Then, the students took a test without the AI and some of the results were quite interesting.
 |
STUDY 01
A request for kindness carried into the next conversation
Researchers gave AI assistants conversations in which a simulated user asked for gentle, supportive responses, then saved the assistants' memories of those conversations. Later, when asked to judge unrelated work, the assistants were more likely to praise weaker poems, and several became less likely to catch mistakes in math. Asking for rigorous criticism could cause a problem in the other direction, with assistants finding fault in sound arguments. The tests showed how a preference learned in one conversation could change the help offered in another, even when that preference no longer fit the task.
Read the study ↗
|
 |
STUDY 02
Keeping track of what the writer actually meant
People wrote social media posts using either a basic AI writing tool or a system called IntentWeaver, which kept their goals visible beside the draft and flagged changes that conflicted with them. With IntentWeaver, writers returned to their larger goals more often and reported noticing more of the moments when their work moved away from those goals. Independent readers also judged the finished posts as better aligned with what the writers wanted to communicate. The posts were clearer, although their overall quality ratings were not significantly higher.
Read the study ↗
|
 |
STUDY 03
They could ask for help but not the answer
In an engineering drawing course, students chose to work without AI, use it freely, or follow a rule that let them ask it to check their own descriptions and reasoning but not give them the answer. Four weeks after the course ended, they took a test they were instructed to complete without AI. The group following the rule averaged 67 out of 100, compared with 45 for unrestricted AI and 49 for no AI. Because the students chose their groups and the test was unsupervised, the study cannot establish that the rule caused the difference.
Read the study ↗
|
Also worth your time
More discussion did not mean more improved essays. Students revising essays with MAIKE, an AI tutor that asked questions without writing for them, spent a larger share of their messages discussing ideas than students using a comparison tutor. About the same proportion in each group improved their essay scores. Read the study
Better code with less time exploring ideas. In short programming tasks, people using an LLM produced more correct and functional code but spent less time generating ideas, with fewer observed creative moments. The finished solutions contained roughly the same number of ideas as those produced without AI. This paper first appeared as a preprint in May. Read the paper
Missing the frustration in a voice. In controlled speech tests, AI voice models usually missed a mildly frustrated tone when the words themselves sounded neutral, even though human listeners could pick up on it. Read the study
The evidence made a difference. In experiments with people who held conspiracy beliefs, removing factual evidence from AI responses substantially weakened their ability to change those beliefs, while several changes to the style of the conversation did not. Read the study
|
SOMETHING TO PLAY
Try The Call
A booking looks wrong or a delivery looks late, and an assistant thinks it knows what happened. Make your first call, open the evidence, and see whether you still agree once you have checked the details. There is no timer, so you can be as suspicious as you like. The fun is finding the thing that changes your mind, or finding out you were right to stick with your first answer.
Play The Call →
|
The Reckoning
I can understand all of the benefits of having an AI that would be able to remember how we like to be treated. It is tiring to explain ourselves over and over, and there is something appealing about opening a conversation with the feeling that whoever, or whatever, we are talking to already knows us. But just because we sometimes want some reassurance after a difficult day doesn't mean we want reassurance all the time, about every single thing. If we have made a mistake in our work today, we probably want somebody to tell us honestly, even if we were asking that same person to be a little kinder to us yesterday.
I think the writing study gets at a similar issue, if we turn to AI for everything. We can spend so much time fixing the words that we wrote that we stop really examining whether they say what we meant, or what we want the reader to take away from it. A sentence can sound better and still leave out the context and depth of the thing we were trying to explain. And if we keep just digging down to try to improve every sentence, then I worry we will end up with something that reads well, but is now nowhere near saying the thing we set out to say.
That is why I liked the rule the engineering students were using in the third study. They had to bring some of their own thinking to the problems before asking AI to check it. If something was missing, they still had to work out on their own how to fix it. The later scores are encouraging, even though we can't know how much of the difference came from the rule and how much came from the students who chose to follow it.
I would like to see more help that leaves us involved like that, rather than just allowing AI to run with the whole process. I think it is perfectly healthy if we ask AI to point out a gap in an argument, or help us understand why an answer is wrong, without asking it to do the whole thing for us. Admittedly, that will take longer, and there are days when we just need something done. But when we do have the time, I think it is worth spending some of it on the parts we want to keep being able to do ourselves, and the things we want to teach others in the future as well.
Your turn
Think about the last time you asked AI for feedback. What did you want it to help you understand, and what kind of response were you hoping to hear? Take a few minutes to write about whether those were the same thing.
Michael McNamara
Illustrations created with AI. They are editorial images, not depictions of research participants.
|