Introduction

In this paper, we propose an effective automatic video colorization method. We first use optical flow to estimate motion vectors for initial matching between the monochrome and colored reference frames. Then an effective colorization method is implemented based on matching results. Besides, we introduce a novel adaptive multi-frame reordering method to obtain the robust colorization results of the whole video sequence.

Experimental Results

The proposed method is implemented by Matlab R2014a platform. Our video colorization method is tested in two ways. One is to use frames randomly selected from some standard video sequences as references to color their next frames in the sequences. Another way is to use one selected frame as a reference and colorize the following few frames. PSNR is computed for objective quality evaluation by our colorization method. We compare our method with Zrihem’s method [1], Veeravasarapu’s method [2] and the baseline made by the optical flow method [3].

Experiment 1:

We use frames randomly selected from some standard video sequences as references to colorize their next frames in the sequences. Fig.1 shows the results.







Zrihem [1]	Veera [2]	Baseline [3]	Proposed

Fig.1 Comparison with different methods on colorizing next frames of randomly selected reference frames.

Experiment 2:

We choose 55 consecutive frames in each sequence and preserve color information every 11 frames to colorize the following 10 frames. Table 1 shows average PSNR values of the results. Fig.2 shows example results of two set of 11 frames in the type of a line chart. Fig.3 shows an example result of the sequences.

Sequence	Zrihem [1]	Veera[2]	Baseline[3]	In-Order	Proposed
Silent	30.47	36.87	41.05	44.68	44.99
Soccer	30.76	35.67	35.56	40.85	41.03
Foreman	30.56	37.51	37.97	41.76	42.50
Tennis	29.49	34.78	35.81	38.44	38.75
MomDaughter	30.88	37.75	36.89	44.40	44.80
Average	30.43	36.52	37.46	42.03	42.41

Table 1. PSNR(dB) comparison of some colorization results from different methods.


(a) Soccer	(b) Foreman

Fig.2 Video colorization results in Soccer and Foreman sequences frame by frame.


Zrihem [1]	Veera [2]	Baseline [3]	In-Order	Proposed

Fig.3 An example video colorization result of different methods by colorizing the following 10 frames with the help of color information of the first frame.

Download

paper

MATLAB implementation code (Dropbox)

Citation

@INPROCEEDINGS{Xia2016VCMRR, author={S. Xia and J. Liu and Y. Fang and W. Yang and Z. Guo}, booktitle={2016 IEEE International Conference on Image Processing (ICIP)}, title={Robust and automatic video colorization via multiframe reordering refinement}, year={2016} }

References

[1] Nir Ben-Zrihem and Lihi Zelnik-Manor, “Approximate nearest neighbor fields in video,” in Proc. IEEE Int’l Conf. Computer Vision and Pattern Recognition, pp. 5233–5242, June 2015.

[2] V S Rao Veeravasarapu and Jayanthi Sivaswamy, “Fast and fully automated video colorization,” in Proc. SPIE Int’l Conf. on Signal Processing and Communications, pp. 1–5, July 2012.

[3] Moritz Menze, Christian Heipke, and Andreas Geiger, “Discrete optimization for optical flow,” in Proc. German Conference on Pattern Recognition, October 2015.

Back to Projects Page