This paper presents a method to synthesize a realistic facial animation of a target person, driven by a facial performance video of another person. Different from traditional facial animation approaches, our system takes advantage of an existing facial performance database of the target person, and generates the final video by retrieving frames from the database that have similar expressions to the input ones. To achieve this we develop an expression similarity metric for accurately measuring the expression difference between two video frames. To enforce temporal coherence, our system employs a shortest path algorithm to choose the optimal image for each frame from a set of candidate frames determined by the similarity metric. Finally, our system adopts an expression mapping method to further minimize the expression difference between the input and retrieved frames. Experimental results show that our system can generate high quality facial animation using the proposed data-driven approach.

[Paper] [Code] [IEEE Trans. MM]

Fig 1. System overview. The optical flow between each query frame and its neutral face is first mapped to the target person, which is used to perform retrieval from the database. Meanwhile, the neutral image of the target person is warped to generate an EMI sequence with the query performance. Finally, the retrieved sequence is refined by the EMI to synthesize the final result.