When a verification or fraud workflow is not sure about something, it sends it to a person. Manual review, a trained analyst looking at the selfie, the document, or the video, is the human backstop almost every system relies on, the place where ambiguous cases go to be judged. Against traditional fraud, a forged document or a mismatched photo, it works. Against synthetic media, it quietly fails, and the failure is dangerous precisely because it is quiet: a convincing deepfake passes manual review not despite being fake but because it is convincing, and the reviewer's clean verdict then functions as false assurance.
This piece explains where manual review fails against synthetic media, and why. It covers the role review is supposed to play, why human perception cannot reliably catch modern fakes, how scale and speed break the model structurally, the attacks that are simply invisible to a reviewer, and the change that actually fixes it. It is written for the fraud, risk, and trust-and-safety leaders who still have a human review queue as their deepfake defense.
The uncomfortable headline is that a clean manual pass is no longer evidence that a file is genuine. Once that is true, review cannot be the control that catches the fake, and the job of the humans has to change.
- Manual review is the standard human backstop in fraud, KYC, and moderation workflows, and against synthetic media it quietly fails.
- Humans detect modern deepfakes at roughly chance, 50 to 60% in studies, and only about 0.1% can reliably tell real from synthetic, so a clean manual pass is not evidence a file is genuine.
- The old visual tells, odd blinking or strange teeth, no longer apply to diffusion-era fakes, leaving reviewers with nothing concrete to judge.
- Scale breaks review structurally: a fraud ring can run thousands of synthetic attempts a day at near-zero cost, while human review scales only with headcount.
- Speed and fatigue make it worse, and injected streams and frequency artifacts are simply invisible to a human reviewer.
- The Arup video-call fraud, a multi-person synthetic meeting with no detection in the loop, shows that even being on the call does not help.
- Regulators now agree: Hong Kong's HKMA requires AI-based deepfake detection for video KYC because manual review cannot meet the volume or accuracy bar.
- The fix is to change what humans are for: automated detection does the finding, and humans adjudicate the hardest flagged cases, aided by explainable evidence.
The Role Manual Review Is Supposed to Play
Manual review earns its place as the human-in-the-loop backstop. Automated rules and checks handle the clear-cut cases, and anything ambiguous, a document that looks slightly off, a selfie the system is unsure about, a transaction that trips a rule, gets routed to a trained analyst to make the call. This works well for the frauds it was designed for, because those frauds leave human-perceptible evidence: a cut-and-paste document edit, a photo that does not match the ID, an inconsistency a careful eye can spot. The entire model rests on an assumption that was true for decades, that a skilled human looking closely can tell a genuine artifact from a tampered one.
Synthetic media breaks that assumption at its foundation. A deepfake is not a clumsy edit with visible seams; it is a wholesale fabrication engineered to look real, and a synthetic identity is internally consistent by construction, with a face, a document, and a story that all agree with each other. There is nothing for the analyst's trained eye to catch, because the thing that would normally betray a fake, a visible flaw, is exactly what the technology is built to eliminate. The reviewer is being asked to do a job the content was designed to defeat.
Why Human Perception Fails Against Synthetic Media
The evidence on this is blunt. Studies put human accuracy at identifying modern deepfakes at roughly chance, in the range of 50 to 60%, with a meta-analysis landing near 55% and dropping further under time pressure. For video specifically, human detection accuracy sits around 57%, statistically indistinguishable from guessing, which means a video-KYC agent on a call is not meaningfully evaluating for synthetic media at all. The starkest figure comes from a 2025 study finding that only about 0.1% of people can reliably distinguish modern synthetic content from real. The practical implication is the one that matters: a clean manual pass is not evidence that a file is genuine.
Two things make this worse than a simple accuracy problem. First, the old heuristics are dead. The advice to look for unnatural blinking, warped teeth, or odd hands worked on 2019-era deepfakes and fails completely against diffusion-era generators, so a reviewer following the familiar checklist is checking for flaws that no longer exist. Second, confidence is miscalibrated: people believe they can tell, which is why the failure goes unnoticed. The canonical illustration is the Arup case, in which an employee joined a video call with what appeared to be the CFO and colleagues, all of them synthetic, and authorized a transfer of tens of millions. Being a human in the loop, on the call, looking at the faces, provided no protection at all.
Why Scale and Speed Break It
Even if human accuracy were better, the economics would still defeat manual review, because the two sides scale completely differently. A fraud ring can generate thousands of synthetic session attempts per day at near-zero marginal cost, while a human review team scales only linearly with headcount. That asymmetry means manual review cannot absorb first contact in a scaled deepfake campaign; the queue simply overflows, and reviewers under pressure get faster and less accurate at exactly the wrong moment. Mobile-first onboarding compounds it, because the same attack is easy to automate and hard to review by hand, and the speed that onboarding needs for conversion leaves little room for careful human inspection in the first place.
Fatigue and consistency finish the job. Human reviewers tire, drift, and grow complacent across a long shift, and two analysts, or the same analyst on two days, will not judge identically. The result in many organizations is review theater: a queue that creates the appearance of a control while letting convincing, high-volume synthetic attacks through. This is why the honest conclusion in the field is that human judgment does not scale as a primary detection layer, and why manual detection is now described not just as insufficient but as introducing liability exposure, since relying on it leaves a documented gap a regulator or a court can point to.
The Invisible-Attack Problem
Beyond perception and scale, some synthetic-media attacks are not detectable by a human reviewer under any conditions, because the evidence is not visible at all. An injection attack feeds a synthetic video stream directly into the verification pipeline through a virtual camera, so there is no physical scene and nothing for a reviewer watching the feed to notice; the manipulation lives below the layer a human can see. The statistical signatures that give a fake away to a detector, frequency-domain fingerprints, the absence of a real physiological pulse, subtle generator artifacts, are by definition imperceptible to the eye. And the signals that expose a coordinated campaign, bulk timing patterns and clusters of attempts sharing infrastructure, are invisible from inside any single session that a reviewer is looking at.
This is the deepest reason manual review fails: it is being asked to find things that leave no human-perceptible trace. A reviewer can still catch a crude document mismatch, but they cannot see an injected stream, read a frequency fingerprint, or perceive a missing pulse, and those are precisely where modern synthetic-media detection operates.
The Fix: Change What Humans Are For
The answer is not to remove humans but to change their job. Manual review should stop being the control that makes first contact with synthetic media and become the adjudication layer that handles the hardest cases after automated detection has done the finding. In a well-designed flow, automated detection screens everything, high-confidence synthetic media is blocked or quarantined automatically, and only genuinely borderline cases are routed to analysts, with the context needed to decide quickly. Human judgment is valuable for ambiguity, escalation, and exception handling; it is not valuable as the thing that detects the fake in the first place, and asking it to be is the mistake.
Two elements make the new arrangement work. The first is that the detection layer reads the signals humans cannot, at a scale and speed humans cannot match, which is the only way to meet an adversary running thousands of attempts a day. The second is explainability: a system that flags a session as likely synthetic should be able to articulate why, both for the audit trail and to calibrate the human decision, so the reviewer adjudicates with evidence rather than guessing. Regulators have reached the same conclusion, with Hong Kong's monetary authority requiring AI-based deepfake detection across remote banking video KYC precisely because manual review cannot meet the volume or the accuracy bar, and supervisory expectations in the US, UK, and EU converging on the view that visual review by staff is not a sufficient control. This is the role DuckDuckGoose, based in Delft, is built to fill: DeepDetector analyzes images and video for the signatures of synthetic media that reviewers cannot see, with explainable output that gives analysts something concrete to act on, which is how one customer, the bank bunq, cut its manual review time dramatically while strengthening rather than weakening its defenses. For the human-accuracy evidence in depth, see our guide on how to spot a deepfake, and for the live-call case, how deepfakes enter video KYC.
Frequently Asked Questions
Can human reviewers detect deepfakes?
Rarely, and not reliably. Studies put human accuracy on modern deepfakes at roughly chance, around 50 to 60%, with video detection near 57% and only about 0.1% of people able to reliably tell modern synthetic media from real. The old visual tells no longer apply to diffusion-era fakes, so a clean manual pass is not evidence that a file is genuine.
Why does manual review fail against synthetic media?
For three compounding reasons: humans cannot reliably perceive modern fakes, which leaves reviewers nothing concrete to judge; the attack volume and speed overwhelm teams that scale only with headcount; and some attacks, like injected camera feeds and frequency-domain artifacts, are invisible to a human under any conditions. A convincing fake passes precisely because it is convincing.
Should we remove human review entirely?
No. The point is to change its role, not eliminate it. Human judgment remains valuable for adjudicating genuinely ambiguous cases, handling exceptions, and escalation. What it should not be is the first-line control that detects synthetic media, because that is the job it cannot do. Automated detection should do the finding, and humans the deciding.
What is review theater?
It is a manual review queue that creates the appearance of a control while failing to catch the threat. Against high-volume, convincing synthetic media, a human queue can look like diligence to auditors and customers while letting fakes through, which provides false assurance and can even increase liability by documenting a gap.
How does scale break manual review?
Through asymmetry. A fraud ring can run thousands of synthetic attempts per day at near-zero marginal cost, while a human team scales only by hiring. The queue cannot keep up, reviewers under pressure get faster and less accurate, and manual review cannot absorb first contact in a scaled campaign, which is why automation has to filter content before humans see it.
Do regulators accept manual review as a deepfake control?
Increasingly not. Hong Kong's monetary authority requires AI-based deepfake detection across remote banking video KYC specifically because manual review cannot meet the volume or accuracy bar, and supervisory expectations in the US, UK, and EU are converging on the view that staff visual review is not a sufficient control against synthetic media.
What is the right way to combine humans and detection?
Automated detection screens everything, blocks or quarantines high-confidence fakes, and routes only borderline cases to analysts, who adjudicate with explainable evidence rather than guessing. This uses machines for what they do well, reading invisible signals at scale, and humans for what they do well, judgment on hard cases, and it is both more accurate and more efficient than either alone.
About the author
