I am building a real-time talking avatar using LiveKit Agents and Wav2Lip.
The goal is to stream audio + lip-synced video to a LiveKit room in real time.

What I am trying to build

Pipeline

User mic
 → STT
 → LLM
 → TTS
 → mel spectrogram chunks
 → Wav2Lip (GPU)
 → video frames
 → LiveKit video track

• Audio is streamed live via LiveKit TTS

• Video frames are generated per mel chunk using Wav2Lip

• A single avatar image (avatar.png) is used as the face input

What I have implemented so far

1. Audio (TTS) node

• I stream audio frames immediately to LiveKit (low latency)

• I buffer PCM audio per response

• After TTS completes, I:

  • Convert PCM → float

  • Resample to 16 kHz

  • Generate mel spectrogram

  • Split mel into chunks (≈ 24 FPS equivalent)

  • Push mel chunks into an async queue for the avatar node

async for frame in Agent.default.tts_node(self, text, model_settings):
    pcm_chunks.append(frame.data)
    yield frame  # audio goes live immediately

2. Avatar (video) node

• A separate async task consumes mel chunks

• For each mel chunk:

  • Run datagen(...)

  • Run Wav2Lip inference on GPU

  • Composite mouth region back into avatar frame

  • Convert frame to RGBA

  • Push frame to LiveKit using VideoSource.capture_frame

Simplified loop:

mel_chunk = await avatar_queue.get()

gen = datagen([avatar_frame], [mel_chunk])
img_batch, mel_batch, frames, coords = next(gen)

pred = model(mel_batch, img_batch)
composite frame
video_source.capture_frame(...)

I limit queue size to avoid memory blowup:

if avatar_queue.qsize() < 5:
    await avatar_queue.put(mel_chunk)

Problems I am facing

1. Avatar video freezes or “sticks”

• Video updates for a few frames

• Then stops updating even though audio continues

• Sometimes resumes later

2. Avatar turns blue

• Random frames appear blue / color-shifted

• I suspect BGR ↔ RGB ↔ RGBA conversion issues

• Or uninitialized frame usage during timeouts

3. Audio breaks / stutters

Logs show:

silero inference is slower than realtime
process memory usage is high

Symptoms:

• Audio drops or breaks mid-sentence

• Video and audio drift apart

What I already tried

• Reduced FPS from 25 → 24

• Limited mel queue size

• Disabled writing video files (pure streaming)

• Moved Wav2Lip to GPU

• Disabled heavy logging

Still seeing freezes and drift.

Please can anyone help me how can I resolve this issue?

I want:

• Smooth real-time lip-sync

• No frame freezes

• Stable audio playback

• Acceptable latency (quality can be slightly reduced)

Please can anyone guide me on proper architecture or timing model.
Thank you!!