Fig.1 An overview of the proposed method. (a) A LR input video sequence. (b) Facial components of each input frame are generated by facial component decomposition. (c) HR frames are reconstructed by neighbor embedding method with locality-constraint, then each frame is processed by Intra and Inter Nonlocal Means method. (The green blocks show the similar patches corresponding to the patch in the red block). (d) The output HR video sequence.

Abstract

In this paper, we propose a novel multi-pose face hallucination method based on Neighbor Embedding for Facial Components (NEFC) to magnify face images with various poses and expressions. To represent the structure of a face, a facial component decomposition is employed on each face image. Then, a neighbor embedding reconstruction method with locality-constraint is performed for each facial component. For the video scenario, we utilize optical flow to locate the position of each patch among the neighboring frames and make use of the Intra and Inter Nonlocal Means method to preserve consistency between neighboring frames. Experimental results evaluate the effectiveness and adaptability of our algorithm. It shows that our method achieves better performance than the state-of-the-art methods, especially on the face images with various poses and expressions.

Paper and Code

[pdf][Code]

Experimental results

Image Face Hallucination

PSNR=28.90	30.38	23.82	29.46	26.01	31.42	-
PSNR=28.85	30.07	20.00	28.02	25.34	30.70	-
PSNR=32.77	34.92	21.64	28.76	-	36.35	-
PSNR=28.78	29.82	22.03	29.58	-	32.78	-
Bicubic	NRSR [1]	EFH [2]	PFH [3]	SFH [4]	Proposed	Ground Truth

The scaling factor is 4. Images are taken from CAS-PEAL-R1 database[5].

Video Face Hallucination

Reference

[1] Y. Li, J. Liu, W. Yang, and Z. Guo, “Neighborhood regression for edge-preserving image super-resolution,” in Proc. IEEE Int’l Conf. Acoustics, Speech, and Signal Processing, 2015.

[2] X. Wang and X. Tang, “Hallucinating face by eigen- transformation,” IEEE Transactions on Systems, Man, and Cybernetics—Part C: Applications and Reviews, vol. 35, no. 3, pp. 425–434, 2005.

[3] X. Ma, J. Zhang, and C. Qi, “Hallucinating face by position-patch,” Pattern Recognition, vol. 43, no. 6, pp. 2224–2236, 2010.

[4] C.-Y. Yang, S. Liu, and M.-H. Yang, “Structured face hallucination,” in Proc. IEEE Int’l Conf. Computer Vi- sion and Pattern Recognition, 2013.

[5] W. Gao, B. Cao, S. Shan, X. Chen, D. Zhou, X. Zhang, and D. Zhao, “The cas-peal large-scale chinese face database and baseline evaluations,” IEEE Transaction- s on Systems, Man, and Cybernetics—Part A: Systems and Humans, vol. 38, no. 1, pp. 149–161, 2008.