You watch a short video of a car approaching an intersection, notice the traffic sign, and see it strike a pedestrian. The next day, you read an AI-generated summary of that video, one that happens to get the sign wrong. Asked afterward what you actually saw, you become measurably less likely to answer correctly. That is the core result of a new study from Georgetown University and the University of Washington: a misleading AI summary can distort people’s memory of an event they witnessed directly, which matters because AI-generated summaries are already spreading into newsrooms, offices, and policing.
AI Summaries Are Already Everywhere
Large language models like ChatGPT and Gemini are now routinely used to condense meeting recordings, news footage, and surveillance video into a few bullet points. Some U.S. police departments have gone further, adopting separate tools that draft incident-report narratives from transcribed body-camera audio. One such product, Axon’s Draft One, generates a draft within minutes of a call ending, which an officer must then review and approve; the Fort Collins, Colorado police department has reported an 82% drop in time spent writing reports after adopting it. That tool drafts text from audio transcripts rather than summarizing video, and it was not tested in this study — but it illustrates how quickly AI-written accounts are entering settings where accuracy carries real stakes.
That broader shift is what worries the study’s authors. “AI is a new method of delivering misinformation,” said lead author Mattea Sim, an assistant research professor at Georgetown’s McCourt School of Public Policy. Co-author Yael Eiger, a Ph.D. candidate at the University of Washington who studies technology in policing, put it more bluntly: she said it concerns her that police departments may already be relying on video-summarization technology without the kind of testing this study attempted.
Testing Memory With a Car-Crash Video
The researchers ran their test in two stages. First, they asked ChatGPT-5.5 and Gemini 2.5 Flash-Lite to summarize two short animated videos of a car-pedestrian collision. The two videos were nearly identical except for one detail: one showed the car approaching a stop sign, the other a yield sign. Each model summarized each video five times, producing 20 summaries in total.
Second, 360 people were recruited to watch one of the two videos themselves. One to two days later, the researchers had each participant read a summary of what they had just seen — either an accurate one or one with a single planted error that reversed the key sign detail — and told them, at random, that the summary had been written by either an AI system or a human transcriber. Of the 360 who started, 328 completed both stages and passed an attention check.
Most Summaries Omitted the Key Event
Before testing human memory, the researchers first graded the 20 AI-written summaries against a list of details the researchers had identified as central to the event. The results were not reassuring: on average, the summaries omitted 51.6% of those central details, and 19 of the 20 summaries failed to mention the single most important one — the car hitting the pedestrian.
The two models differed in other ways. Every summary from both ChatGPT and Gemini contained at least one error. Gemini’s summaries included a critical inaccuracy — one that reversed a core fact, such as the traffic sign, rather than just dropping a minor detail — in all of its summaries, compared with 30% for ChatGPT. Both models also occasionally invented details that were never in the video at all, a pattern researchers call hallucination. These figures come from a small, controlled test of two models summarizing two short videos, so they should be read as evidence that serious errors can occur rather than as a general failure rate for AI video summaries.
“AI has the potential to generate misinformation, even absent any adversarial intent,” the study authors write.
A Wrong Summary Cut Recall Nearly in Half
The second stage showed these errors don’t stay confined to the summary — they appear to bleed into people’s own memory of the event. Participants who read an accurate summary correctly identified the sign 83.6% of the time. Among those who read the summary with the planted error, accuracy dropped to 44.8%, a gap the researchers found to be statistically significant and unlikely to be explained by chance alone.
That 38.8-percentage-point gap is the study’s central finding: a single misleading detail, embedded in an otherwise ordinary-looking summary, substantially reduced participants’ ability to correctly recall a detail they had directly observed just one or two days earlier. The experiment shows a clear misinformation effect; it doesn’t by itself prove that the original memory was erased or permanently replaced, only that recall of this one detail became much less reliable.
Source Labels Alone Didn’t Help
One of the more striking results is what didn’t matter. Participants told the summary came from an AI were no more accurate, and no less likely to be misled, than those told a human wrote it: 46.3% versus 43.4%, a difference too small to be statistically meaningful. How much participants said they trusted AI in general, or how often they used it, also made no measurable difference.
In this experiment, simply labeling a summary as AI-generated rather than human-written did not reduce the misleading effect. The researchers say this challenges a common assumption in AI-safety discussions — that keeping a “human in the loop,” meaning a person who reviews the AI’s output, is enough to catch its mistakes. If a misleading summary quietly shapes the reviewer’s own recollection of the source material, that reviewer may be less able to notice anything is wrong. The study did not test other safeguards, such as requiring reviewers to check specific claims against the original recording, so it can’t yet say which interventions would actually help.
Why This Matters for Courts and Policing
The experiment used an animated video, student volunteers, and a lab setting — not real police officers, real witnesses, or an actual criminal case. Its relevance to policing is more general: when an AI-generated account becomes part of how someone reconstructs what happened, errors in that account may shape what they later remember. If an officer or witness reads an inaccurate AI-generated recap of body-camera footage before giving a formal statement, that recap could influence their later recollection — a real concern, though not one this particular study tested directly.
This isn’t only a human blind spot, either. A separate study covered previously on this site found that AI vision systems fail to reproduce the same motion illusions that trick the human eye — one more reminder that machine and human perception can diverge in either direction, so neither can simply be trusted to check the other without independent verification (see our earlier coverage of AI vision and motion illusions).
A condensed version of this research was presented this week at the Ninth AAAI/ACM Conference on AI, Ethics, and Society (AIES 2026), held October 12–14 in Malmö, Sweden, after first appearing as a preprint in September. The authors aren’t calling for AI summaries to be banned from sensitive uses, but they argue error rates like these need to be measured and disclosed before such tools are trusted with decisions that affect legal outcomes.
The researchers are also upfront about what they didn’t test. They used only consumer-facing versions of ChatGPT and Gemini, not the enterprise or custom-tuned models that vendors like Axon actually deploy, and the AI error-rate analysis covered just two short animated videos. Eiger says that gap is itself part of the problem: outside researchers generally have little visibility into which exact models or settings police vendors use, which makes independent auditing difficult. Sim says a natural next step is to study real body-camera footage and real police reports directly, to see whether AI-generated summaries measurably shift how people judge a police-civilian interaction — not just a simple traffic-sign detail in an animated clip.
Xsciens Editorial Note
The finding is not simply that “AI makes mistakes”; it is that a single wrong detail can slip into a person’s memory and quietly bend the judgment of the human who is supposed to be checking.
- Read the 38.8-point gap in context. Accuracy was 83.6% after an accurate summary and 44.8% after a misleading one. But this comes from two short animated videos, two AI models, and 328 analyzed participants, so it is not an error rate for AI summaries in general.
- An “AI-written” label did not help. Accuracy was 46.3% when people were told an AI wrote the summary and 43.4% when told a human did, with no meaningful difference. The study did not test checking steps such as comparing against the original video, so it does not show that human review is useless, only that review has to be designed carefully.
- Not a test of real police tools. The participants were volunteers watching animations, not officers or witnesses in real cases. The classic Loftus and Palmer experiments had already shown that the wording of a question can change what people remember, so the new element here is the AI-generated text as the source of the distortion.
- Our imagination (speculation) — We wonder whether future AI summaries that show the exact moment of the video each sentence is based on, so you can jump to it in one click, might make errors easier to catch. A meeting recording that lets you tap a line and hear the original remark is one example. But if people skip the check even when the evidence is one tap away, the same problem could remain, so this is only our speculation.
References for this note: Georgetown University press release, the paper (arXiv:2609.28820), Loftus & Palmer (1974), explained (SimplyPsychology)
Sources: Georgetown University, “New Research Finds That Misleading AI-Generated Summaries Can Distort Human Memory”, Sim, Eiger & Kohno, “AI-Enabled Human Memory Manipulation” (arXiv:2609.28820), Axon, “Axon Reimagines Report Writing With Draft One” (via Officer.com)
See also: our explainer on cognitive offloading, the habit of handing memory and thinking to tools.