An Agentic Multi-Modal Evaluation Framework for Audio-Driven Generative Video Synthesis: Bridging Semantic Coherence and Quality Alignment

Abstract

Evaluating audio-driven talking-face video requires distinguishing synchronization, visual quality, naturalness, and emotional consistency. This paper develops an evaluation protocol and examines its statistical behavior using a reproducible synthetic dataset. The dataset contains 600 video-level records arranged as 200 source groups with three condition labels, together with ratings from 24 simulated raters. No generated videos, human participants, or model inference outputs are used in the reported analysis. Under the specified simulation, LSE-C correlates with simulated lip-sync scores at ρ = 0.626 and with the holistic composite at ρ = 0.253. The DOVER-labeled variable reaches ρ = 0.744 on the holistic composite, while out-of-fold ridge predictions reach ρ = 0.821. Simulated reference-guided scores show higher agreement than absolute scores, and one simulated judge exhibits substantial order sensitivity. These patterns reflect assumptions imposed by the data generator, including condition effects, reduced reference-protocol noise, and position bias. They illustrate how pooled correlations, metric conflicts, and pairwise agreement can be analyzed; they do not establish the relative performance of the named systems. The resulting protocol specifies the data and validation needed for an empirical study with real videos and human ratings.