Abstract

Visual dubbing aims to modify facial expressions to "lip-sync" a new audio track. While person-generic talking head generation methods have made significant progress in expressive lip synchronization across arbitrary identities, they usually lack person-specific details and fail to generate high-quality results. On the other hand, person-specific approaches enable realistic identity preservation and high lip-sync quality but require extensive training, limiting their adaptability in real-world applications.

Our method combines the strengths of both approaches to generate balanced results in lip synchronization and visual quality while achieving training efficiency. To this end, our pipeline incorporates a virtual dubber, a person-generic talking head, as an intermediate representation. This simplifies identity swapping, enhances efficiency, and improves both visual quality and expression accuracy.

Key innovations include full-head identity swapping and reenactment, eliminating artifacts such as the double chin effect while ensuring temporal stability. Through extensive quantitative and qualitative evaluations, we demonstrate that our approach achieves a superior balance between lip-sync accuracy and realistic facial reenactment.

Furthermore, we validate the robustness of our method with experiments in challenging real-world scenarios, including tilted head poses and facial occlusions. Notably, our pipeline operates effectively with short video clips, emphasizing its efficiency and practicality.