A real-time lip-sync demo built with Three.js, React Three Fiber, and the Web Audio API. A 3D avatar animates its mouth and face in sync with audio playback, driven by real-time viseme detection and emotion expression blending.
Built as an exploration of audio-driven facial animation in the browser — specifically, whether a per-frame viseme pipeline could make a skinned 3D character speak convincingly without a single pre-baked animation frame. Solo project, open source (MIT): one audio track and one custom avatar rig with 50+ ARKit morph targets, deployed live at realtime-lipsync.anjana784.dev.
The challenge
Make a 3D avatar's face convincingly track spoken audio in real time — the mouth, the eyes, and the expression — without animating anything by hand.
The pipeline had to run inside a requestAnimationFrame loop: analyze the current audio buffer, detect the active viseme, and blend it onto the right subset of 50+ morph target influences — without visible snapping, and without the emotion system clobbering the mouth shapes.
The obvious approach — keyframed mouth animation synced to a known track — breaks the moment the audio changes, and says nothing about live speech. The other obvious approach — snapping morph target influences straight to the detected viseme each frame — produces robotic mouth jumps, because viseme detection is frame-sparse and the target values are extremes.
The approach
Separate the concerns. Audio analysis, emotion state, and the render loop each own one thing and communicate through refs — React never re-renders; everything is applied inside the avatar's useFrame.
Two independent lerp engines (one for visemes, one for emotions) both write into the same morphTargetInfluences array, each eased with an ease-in-out curve at a speed of ~12 per second, so mouth shapes and expressions crossfade rather than pop. The blink cycle runs as a separate state machine layered on top with a max-overlay strategy — it takes the maximum of its own value and the underlying influence, so a blink never destroys a smile or an open jaw.
The viseme-to-morph mapping tables and the emotion presets are hand-tuned for this specific model and track. The pipeline generalizes — any GLB with ARKit morph targets works — but the tuning doesn't. And wawa-lipsync is a young library (v0.0.1), so the audio-analysis layer is a dependency I may need to replace as it matures.
Technical details
Audio-to-viseme pipeline
A Lipsync instance from wawa-lipsync is connected directly to the HTMLAudioElement. Each animation frame calls processAudio(), which analyzes the current audio buffer and exposes the active viseme; the result is read into a ref that the avatar consumes. The entire pipeline runs in the browser — no server, no pre-analysis.
Morph target blending
15 visemes map to blend presets in VISEME_TO_MORPH_TARGETS (bilabial contact for P/B/M, funnel for CH/J/SH, stretched smile for the I sound), and 7 emotions map to their own presets in EMOTION_TO_MORPH_TARGETS — each across the model's 50+ ARKit morph targets. Both systems lerp toward their presets every frame with an ease-in-out-shaped step, so transitions between sounds and expressions are smooth rather than instantaneous.
Emotion timeline
Because the demo plays a single known track, the emotion sequence is derived rather than authored: buildEmotionTimeLine divides the audio duration into equal segments (happy → neutral → surprised → thinking → happy), and a rAF loop in useEmotionSync samples audio.currentTime against those segments to pick the active emotion. The lerp engine then crossfades the face between expressions.
Blink state machine
When the avatar is idle, a three-state blink machine (closing → opening → idle) drives the eyelids with smoothstep easing and a randomized interval of 2.2–6 seconds. Eye values are applied with a max-overlay strategy against whatever the emotion or viseme systems wrote, so blinking layers over a smile or a surprised brow without resetting the underlying expression.
Scene & shaders
The scene is a React Three Fiber Canvas with depth fog tied to the background. Two custom GLSL shaders handle the presentation: a radial gradient backdrop that fades to black at its edges, and a soft ambient shadow disc under the character that darkens toward its center. Everything else — lights, ground plane, camera — is plain Three.js.
Debug & UX
A Leva panel lets you drive individual visemes and emotions manually for tuning. The shipped page layers a loading overlay (waits for the GLB model, the audio, and the lip-sync manager), a real-time tech HUD showing the current viseme, emotion, FPS, and active morph targets, and a help overlay with keyboard shortcuts (space/K to play, ? for help).
Outcome
Shipped live at realtime-lipsync.anjana784.dev between October 2025 and July 2026 — open source (MIT), with the full viseme and emotion reference tables documented in the README.
The core deliverable is the mapping itself: a documented, reusable bridge from wawa-lipsync visemes to ARKit morph targets — 15 viseme presets and 7 emotion presets across 50+ morph targets — plus a blend engine that makes the result look intentional instead of mechanical.
Reflection
Separating the viseme and emotion systems paid off — each could be tuned in isolation, and in real time via the Leva panel, without touching the other. The max-overlay blink strategy was the smallest piece of code in the project and the one that sold the realism.
The emotion timeline is computed by dividing the track into equal segments rather than authored against the actual content — it works, but emotions land at arbitrary moments, and for real dialogue I'd author the segments against the lines. I'd also have fixed the MORPH_TARGETS typo that still sits in lib/utils/types.ts.
The pattern of per-frame state flowing through refs — analysis hooks write, the avatar's useFrame reads, React never re-renders — is now my default architecture for real-time Three.js work in React.