How Face Reenactment and Puppeteering Work

Face reenactment is a deepfake technique that transfers one person's facial expressions, head pose, and mouth movements onto another person's face while leaving the target's identity intact. Unlike a face swap, which changes who you are looking at, reenactment changes what that person appears to be doing and saying. When a live human “driver” controls a target face in real time, answering questions, nodding, reacting on a video call, that real-time mode is what researchers call puppeteering (or puppet-mastery).

This guide explains how both work under the hood: the pipeline they share, the two main technical families (3D-model graphics and neural warping), the newer diffusion-based talking-head systems, and, because this matters most for anyone defending an identity flow, why reenactment leaves a different forensic trail than a face swap, and which signals actually expose it.

It is written for security leads, fraud and compliance officers, and product managers at IDV companies who need a working mental model of the technique, not a research-lab deep dive.

‍

What Is Face Reenactment?

Face reenactment, also called expression reenactment or facial reenactment, is a face-synthesis task in which the facial expressions, head movements, and lip motion of a driving (source) person are transferred onto a target identity, while preserving the target's appearance and identity (D2Fusion survey). The target keeps their face; only their motion is borrowed from someone else.

That single distinction is what separates reenactment from the other major families of facial deepfakes. A comprehensive review of face deepfakes sorts manipulations into four groups: synthesising an entire non-existent face, identity swap (face swapping), attribute manipulation (changing hair, glasses, age), and expression swap. Reenactment sits squarely in the expression-swap category, because it is not designed to change a person's identity at all (Face Deepfakes review). The word “reenactment” is literal: it brings a still image or a target video to life by driving it with the head, lip, and expression movements captured from a driving video.

The technique has legitimate uses, including film dubbing, low-bandwidth video conferencing, accessibility avatars, and visual effects, but the same capability is what makes it a fraud tool. Because the target's identity is preserved, a convincing reenactment can put words in a real person's mouth using nothing more than their existing likeness.

‍

What Is Puppeteering?

Puppeteering, or puppet-mastery, is reenactment operated like a marionette: a performer sits in front of a camera and their motion and expressions drive the deformation of a face appearing in a target image or video in real time (face swap vs. reenactment overview). In the research literature the two terms are often used interchangeably, since puppet-mastery is essentially the live, performer-controlled application of face reenactment.

The practical difference is timing and control. Pre-rendered reenactment produces a finished clip. Puppeteering runs the same machinery interactively, so the driver can respond to a question, follow a “turn your head” prompt, or react to something said on a call. A closely related variant is lip-sync (audio-driven) reenactment, where the mouth region, or an entire talking head built from a single photo, is driven by an arbitrary audio recording rather than a driving video. That audio-driven mode is what pairs naturally with voice cloning to produce a complete synthetic persona that both looks and sounds like a target.

For a defender, puppeteering is the more dangerous mode for one simple reason: it works in live, two-way conversations, which is exactly where most “is this person real?” trust decisions still get made. (We cover how this differs from camera-injection threats in our breakdown of presentation attacks vs. injection attacks.)

‍

How Face Reenactment Works: The Shared Pipeline

Whatever the underlying model, nearly every reenactment system follows the same four-stage logic. Understanding these stages is what lets you reason about where the artifacts come from.

‍

‍

1. Represent the target's appearance. The system needs a representation of the target identity, extracted either from a video or, increasingly, from a single image. Modern “one-shot” methods need only one reference photo, which is why a publicly posted headshot is enough raw material.
   ‍
2. Extract motion from the driving video. The driver's expressions and head pose are captured as a compact motion code, encoded as 3D model parameters, facial landmarks, or self-learned keypoints.
   ‍
3. Disentangle identity from motion, then transfer. The model must separate who the target is (appearance) from what the driver is doing (motion), then apply the driver's motion to the target's appearance without dragging the driver's identity along with it.
   ‍
4. Render and blend the output. A renderer or neural generator produces the final frames, fills in regions that become newly visible as the head moves (disocclusion), and blends everything into a coherent image. This generation step is where most visible and statistical artifacts are introduced.

The differences between methods come down to how each stage is implemented, and that splits into two main technical families, plus a fast-growing third.

‍

Family 1: 3D Model / Graphics-Based Reenactment

The graphics approach reconstructs an explicit 3D model of the head, typically a 3D Morphable Model (3DMM) that parameterises identity, expression, and pose, sometimes layered with the Facial Action Coding System (FACS). The landmark example is Face2Face, which builds a 3D model of both the source and target faces, computes the source's expression parameters, transfers them to the target model, then re-renders the result. It enforces dense photometric consistency, minimising the error between the rendered face and the real input frame, so the synthetic output stays very close to a genuine image, which is part of what makes it hard to spot (Understanding the Security of Deepfake Detection).

The strength of this family is precise, explicit control over geometry and head pose, and it can run in real time from a commodity webcam. Its weakness is that it depends on a good 3D fit; errors in the reconstruction show up as subtle geometric or shading mismatches.

‍

Family 2: Neural Warping (Keypoint and Flow-Based)

The learning-based family skips the explicit 3D rig and instead warps the source image to match the driver's motion. The reference architecture is the First Order Motion Model (FOMM) and its lineage (Monkey-Net, then FOMM, then face-vid2vid, then LivePortrait). FOMM decouples appearance and motion through a self-supervised formulation built from four components (talking-head generation survey; low-bandwidth video-chat method):

• A keypoint detector learns around ten keypoints, with no manual labels, for both the source and driving frames. These are not anatomical facial landmarks; they are learned points that capture motion.
• A dense motion network turns the sparse keypoint differences into a dense optical-flow field plus an occlusion map.
• The flow field warps the source image's features using a differentiable sampling operation, and the occlusion map masks regions the warp cannot explain.
• A generator (decoder) reconstructs the final frame, hallucinating any newly revealed regions that the warp could not supply.

This family's advantage is that it animates a target from a single image with no 3D model and runs fast enough for real time. Its weakness is structural: warping struggles with large pose changes, and the generator has to invent content for occluded areas, which is precisely where it leaves tells. LivePortrait, introduced in 2024, is a current real-time example of this family, using implicit keypoints plus lightweight stitching and retargeting modules and trained on more than 69 million frames (TalkingHeadBench).

‍

Family 3: Diffusion and Neural Rendering

The newest wave replaces or augments the GAN generator with diffusion models and volumetric/NeRF-style rendering, in systems such as X-Portrait, EMOPortraits, VOODOO 3D/XP, and Ditto. These push fidelity higher, handle expression and pose more robustly, and increasingly support audio-driven generation, which is the engine behind realistic one-shot talking heads (TalkingHeadBench). For defenders, the takeaway is that the artifact profile keeps shifting: each new generator family removes some of the tells the previous one left behind.

‍

Why Reenactment Is Uniquely Dangerous

Reenactment and puppeteering escalate the deepfake threat along two axes that face swapping does not.

It needs almost no data. One-shot methods animate a target from a single reference image. There is no need to collect hours of footage or train a per-target model; a scraped profile photo is sufficient raw material.

It works live. Because puppeteering runs interactively, an attacker can hold a two-way conversation as the target, the single capability that defeats most human verification habits. The 2024 Arup case in Hong Kong is the canonical example: attackers used AI-generated video and audio replicas of the CFO and other executives in a live multi-person video conference, not a pre-recorded clip, and a finance employee, seeing and hearing his “superiors,” authorised a transfer that ultimately cost the firm roughly USD 25 million (Security Today). A near-identical playbook hit a Singapore firm in March 2025, where attackers proactively suggested a video call to overcome the victim's skepticism (Brightside AI).

The scale is no longer marginal. Veriff's 2025 Identity Fraud Report found that roughly 1 in 20 identity-verification failures is now linked to deepfakes, as fraudsters manipulate live video to bypass verification (Veriff). And the reason these attacks work is not technical naivety; it is that humans are poor detectors. People correctly identify high-quality deepfake video in only about 24.5% of cases (Security Today). Live puppeteering also dismantles the classic controls built for business email compromise: a phone callback does not help when the voice is cloned, and a video confirmation does not help when the face is puppeteered.

‍

How to Detect Reenactment: Six Signal Families

Here is the detail most defenders miss. Many face-swap detectors hunt for a blending boundary, the seam where a swapped face is composited onto a target head. Reenactment usually has no such seam, because the identity is never replaced; the whole face region is warped or re-synthesised. So the signals that catch reenactment are different, and they lean heavily on motion over time rather than a single frame.

‍

Temporal and spatiotemporal inconsistency. Frame-by-frame generation often fails to produce a temporally coherent sequence, leaving flicker, jitter, or warping artifacts that only appear across frames. Detectors using 3D CNNs, CNN+LSTM stacks, and spatiotemporal-inconsistency learning target exactly this (Face Forgery Detection survey).

Lip and mouth semantics. Approaches like LipForensics focus on high-level irregularities in mouth movement, which is especially effective against lip-sync and audio-driven reenactment (Face Forgery Detection survey).

Optical-flow inter-frame dissimilarity. Measuring motion between consecutive frames with optical flow can separate reenacted video from genuine footage, because warped motion fields behave differently from real ones.

Biomechanical / kinematic correlation. A real face is a coupled physical system: eyebrow and eyelid, jaw and lips move in correlated ways. Models that render facial parts semi-independently can produce motion that looks plausible in isolation but is subtly uncorrelated between parts. Newer detectors explicitly hunt these kinematic inconsistencies, which survive even when there is no obvious flicker (Beyond Flicker).

Frequency-domain artifacts. The upsampling and generation steps in neural generators leave statistical fingerprints in the frequency spectrum that are invisible to the eye but detectable to a model.

Audio-visual synchronisation. For talking-head and lip-sync fakes, mismatches between the audio track and the visible mouth shape are a strong, generator-agnostic tell, and they require analysing video and audio together.

Independent market analyst Mordor Intelligence names DuckDuckGoose AI among the major companies in the fake-image-detection market, alongside larger incumbents, a reminder that specialist detection has become its own category rather than a feature bolted onto verification. For how these signals translate into real-world accuracy, see our analysis of deepfake detection accuracy.

‍

Common Mistakes Defenders Make

Assuming a face-swap detector covers reenactment. It often does not. A model trained to find blending seams has little to grab onto when the identity was never swapped. Reenactment requires temporal and semantic signals, not just spatial ones.

Relying on human review. At a roughly 24.5% identification rate for high-quality video, human “gut checks” are close to a coin flip and worse; they create false confidence.

Single-frame-only detection. The strongest tells in reenactment live between frames. A detector that scores each frame independently throws away the temporal evidence that matters most.

Trusting liveness prompts alone. Puppeteering can follow a “blink” or “turn your head” instruction because a live human is driving it, and injection attacks can bypass the camera entirely. Liveness is necessary but not sufficient.

Training on one generator family. A detector tuned to GAN-warping artifacts can degrade sharply on diffusion-based output. Generalisation across unseen methods, not lab accuracy on one dataset, is the metric that predicts field performance.

‍

Practical Recommendations

Detect across modalities. Because puppeteering usually pairs reenacted video with cloned audio, analysing video and audio together closes a gap that a video-only or audio-only tool leaves open. Weight temporal, frequency, and semantic signals, not just spatial ones, so you catch the seam-free fakes that reenactment produces. Favour detection that generalises across generator families over a model that posts a high score on a single benchmark. And place detection upstream, at identity creation and during live sessions, rather than treating it as a post-incident forensic step.

This is the design philosophy behind DuckDuckGoose's DeepDetector, which combines forensic image and video analysis with explainable output (it shows why a sample was flagged, not just a score) and is built to integrate into IDV pipelines so manipulated media is caught before an account is activated. Keep out-of-band process controls, including code words, four-eyes approval, and independent callbacks on a known number, as a backstop, since they fail differently than any single model. For the broader picture, our deepfake statistics roundup tracks how fast these attack volumes are growing.

‍

Frequently Asked Questions

What is the difference between face reenactment and face swapping?
Face swapping replaces the identity by putting one person's face onto another person's head while keeping the original head's expressions and pose. Face reenactment does the opposite: it keeps the target's identity and appearance but drives their expressions, head pose, and mouth movements from a different person. Swapping changes who; reenactment changes what they appear to do.

What is puppeteering in deepfakes?
Puppeteering, or puppet-mastery, is face reenactment run in real time, where a live performer's movements drive a target face like a marionette. It is the technique behind live deepfake video calls, because the driver can respond to questions and follow prompts interactively rather than playing a pre-recorded clip.

How much data does face reenactment need?
Modern one-shot methods can animate a target from a single reference image. That is a key reason reenactment is dangerous: a publicly available headshot can be enough raw material, with no need to collect extensive footage of the target.

Does face reenactment leave a blending boundary like a face swap?
Usually not. Because the identity is never composited onto a different head, there is typically no seam between a swapped face and the surrounding image. This is why detectors built to find blending boundaries often miss reenactment, and why temporal and semantic signals matter more.

Can deepfake video calls be detected in real time?
Yes, though it is harder than analysing a saved file. Real-time detection looks for temporal inconsistencies, warping and frequency artifacts, and audio-visual desynchronisation during the session. Combining automated detection with out-of-band verification (such as a callback to a known number) is the most reliable approach.

Why can't humans reliably spot reenactment deepfakes?
High-quality synthetic video is correctly identified by people only about a quarter of the time. The artifacts that betray reenactment, such as subtle inter-frame inconsistencies, frequency-domain fingerprints, and uncorrelated micro-movements, are largely invisible to the human eye, which is why automated detection is necessary.

What is the First Order Motion Model?
The First Order Motion Model (FOMM) is an influential neural reenactment architecture that animates a still source image to follow a driving video. It uses self-learned keypoints, a dense motion network that produces an optical-flow field and occlusion map, and a generator that warps and reconstructs the output. Many modern one-shot animation tools build on this lineage.

Are diffusion models used for face reenactment now?
Increasingly, yes. Diffusion-based and neural-rendering systems are pushing reenactment fidelity higher and improving audio-driven talking heads. For defenders this means the artifact profile keeps shifting, so detectors must be evaluated on their ability to generalise to unseen generator families.