This Week's Reckoning: Can AI Get to Know Us and Still Get Us Wrong?


Reckonings ISSUE 17
NOVEMBER 10, 2026

FROM THE QUIET COST

Can AI Get to Know Us and Still Get Us Wrong?

When we ask someone for help, it is not always just the actual help with that project that we are looking for. Sometimes we have had a bad day and just want to have somebody to talk about it with, while we keep our hands busy. Other times, we really do need help with a specific thing, and we are looking for them to show us how to do it. Sometimes, we want to hear about or see where we fell short, and other times, it’s the last thing we want to hear about when we are getting that help. And it can be the same person we go to in both cases, because we trust that they know us well enough to understand when we need one thing, and not the other.

The studies this week made me think about a world where we go to AI for help, more than we go to people, and whether AI will be able to tell the difference and adjust accordingly. One study looked at how an AI’s memory of someone who asked it to be kind changed the feedback it gave that person on unrelated work later. Another study gave writers an AI tool to see if it would help or hurt their ability to write more clearly, and closer to the original goals that they had when they set out to write the post. And in a third study, engineering students used AI to check their thinking, or their process, but could not use it to ask for the answer itself. Then, the students took a test without the AI and some of the results were quite interesting.

Also worth your time

More discussion did not mean more improved essays. Students revising essays with MAIKE, an AI tutor that asked questions without writing for them, spent a larger share of their messages discussing ideas than students using a comparison tutor. About the same proportion in each group improved their essay scores. Read the study

Better code with less time exploring ideas. In short programming tasks, people using an LLM produced more correct and functional code but spent less time generating ideas, with fewer observed creative moments. The finished solutions contained roughly the same number of ideas as those produced without AI. This paper first appeared as a preprint in May. Read the paper

Missing the frustration in a voice. In controlled speech tests, AI voice models usually missed a mildly frustrated tone when the words themselves sounded neutral, even though human listeners could pick up on it. Read the study

The evidence made a difference. In experiments with people who held conspiracy beliefs, removing factual evidence from AI responses substantially weakened their ability to change those beliefs, while several changes to the style of the conversation did not. Read the study

SOMETHING TO PLAY

Try The Call

A booking looks wrong or a delivery looks late, and an assistant thinks it knows what happened. Make your first call, open the evidence, and see whether you still agree once you have checked the details. There is no timer, so you can be as suspicious as you like. The fun is finding the thing that changes your mind, or finding out you were right to stick with your first answer.

Play The Call →

The Reckoning

I can understand all of the benefits of having an AI that would be able to remember how we like to be treated. It is tiring to explain ourselves over and over, and there is something appealing about opening a conversation with the feeling that whoever, or whatever, we are talking to already knows us. But just because we sometimes want some reassurance after a difficult day doesn't mean we want reassurance all the time, about every single thing. If we have made a mistake in our work today, we probably want somebody to tell us honestly, even if we were asking that same person to be a little kinder to us yesterday.

I think the writing study gets at a similar issue, if we turn to AI for everything. We can spend so much time fixing the words that we wrote that we stop really examining whether they say what we meant, or what we want the reader to take away from it. A sentence can sound better and still leave out the context and depth of the thing we were trying to explain. And if we keep just digging down to try to improve every sentence, then I worry we will end up with something that reads well, but is now nowhere near saying the thing we set out to say.

That is why I liked the rule the engineering students were using in the third study. They had to bring some of their own thinking to the problems before asking AI to check it. If something was missing, they still had to work out on their own how to fix it. The later scores are encouraging, even though we can't know how much of the difference came from the rule and how much came from the students who chose to follow it.

I would like to see more help that leaves us involved like that, rather than just allowing AI to run with the whole process. I think it is perfectly healthy if we ask AI to point out a gap in an argument, or help us understand why an answer is wrong, without asking it to do the whole thing for us. Admittedly, that will take longer, and there are days when we just need something done. But when we do have the time, I think it is worth spending some of it on the parts we want to keep being able to do ourselves, and the things we want to teach others in the future as well.

Your turn

Think about the last time you asked AI for feedback. What did you want it to help you understand, and what kind of response were you hoping to hear? Take a few minutes to write about whether those were the same thing.

Michael McNamara

Illustrations created with AI. They are editorial images, not depictions of research participants.

Reckonings

Each week, the most revealing AI stories and studies, what they could mean for you and society, and practical games and exercises to help protect the human capacities we do not want to lose.

Read more from Reckonings
A person choosing familiar documents while other sources remain unopened beside a reassuring laptop.

Reckonings ISSUE 18NOVEMBER 17, 2026 FROM THE QUIET COST When AI Agrees With Us, Do We Stop Looking? Whenever we spend multiple days on one particular thing, it is such a good feeling when someone looks at it and tells us that they agree. Maybe we are trying to convince our boss to change the metrics we look at to measure success, and we ask a co-worker whether our argument holds up before we put it in front of them. But the flip side is that the more we work on something, the less willing we...

A woman reading an online discussion with human portraits and a chat bubble on her laptop.

Reckonings ISSUE 16NOVEMBER 3, 2026 FROM THE QUIET COST Can AI Help Without Taking Our Place? When someone posts in an online support group, most of us would agree that getting a quick reply is a good thing. And it probably is, especially in the cases where a person has been sitting with something they haven't been able to tell anyone else in their real life. But there are other people in that group and those people’s comments (or lack thereof) are often impacted by the initial comments made...

A woman arranges her writing ideas on paper cards beside a laptop.

Reckonings ISSUE 15OCTOBER 27, 2026 FROM THE QUIET COST Should AI Make Us Work a Little Harder? We all understand that people are moving to AI more often in order to get things done. Usually, it’s the trivial tasks or the arduous tasks that they don’t enjoy. But the irony is that a lot of people feel like they are working longer and harder in the age of AI. So, what was the point of creating it, if all it makes us do is work more? And even if the workload feels lighter, we end up losing the...