Video matting

Video matting is a technique for separating the video into two or more layers, usually foreground and background, and generating alpha mattes which determine blending of the layers. The technique is very popular in video editing because it allows to substitute the background, or process the layers individually.

Video matting methods

Problem definition

When combining two images the alpha matte is utilized, also known as the transparency map. In the case of digital video, the alpha matte is a sequence of images. The matte can serve as a binary mask, defining which of the image parts are visible. In a more complicated case it enables smooth blending of the images, the alpha matte is used as the transparency map of the top image. Film production has known alpha matting since the very creation of filmmaking. The mattes were drawn by hand. Nowadays, the process can be automatized with computer algorithms.

Left to right: input image, background, foreground, and alpha matte.

The basic matting problem is defined as following: given an image $I$ , compute the foreground $F$ , background $B$ and alpha matte $A$ , such that the equation $I=AF+(1-A)B$ holds true. This equation has trivial solution $A=1$ , $F=I$ , $B$ is any image. Thus, usually an additional trimap must be provided as input. The trimap specifies background, foreground, and uncertain pixels, which will be decomposed into foreground and background by the matting method.

The main criteria for video matting methods from a user perspective are following:

Accurate edge processing
Time stability
Minimal user intervention

Methods description

The first known video matting method ^[1] was developed in 2001. The method utilizes optical flow for trimap propagation and a Bayesian image matting technique which is applied to each image separately.

Video SnapCut,^[2] which later was incorporated in Adobe After Effects as Roto Brush tool, was developed in 2009. The method makes use of local classifiers for binary image segmentation near the target object's boundary. The results of the segmentation are propagated to the next frame using optical flow, and an image matting algorithm ^[3] is applied.

A method ^[4] from 2011 was also included in Adobe After Effects as Refine Edge tool. The propagation of trimap with optical flow was enhanced with control points along the object edge. The method uses per-image matting, but temporal coherence was improved with a temporal filter.

Finally, a deep learning method ^[5] was developed for image matting in 2017. It overcomes most traditional methods.^[6]

Benchmarking

Video matting is a rapidly-evolving field with many practical applications. However, in order to compare the quality of the methods, they must be tested on a benchmark. The benchmark consists of a dataset with test sequences and a result comparison methodology. Currently there exists one major video matting online benchmark,^[6] which uses chroma keying and stop motion for ground truth estimation. After method submission, the rating for each method is derived from objective metrics. As objective metrics do not represent human perception of quality, a subjective survey is necessary to provide adequate comparison.

Top 5 video matting methods ^[6]
Method	Year of development	Ranking place
Deep Image Matting ^[1]	2016	1
Self-Adaptive ^[7]	2016	2
Learning Based ^[8]	2009	3
Sparse Sampling ^[9]	2016	4
Closed Form ^[3]	2008	5

Practical use

Object cutout

Video matting methods are required in video editing software. The most common application is cutting out and transferring an object into another scene. The tool allows users to cut out a moving object by interactively painting areas that must or must not belong to the object, or specifying complete trimaps as input. There are several software implementations:

An interactive video cutout system ^[10]
Adobe After Effects Rotobrush tool ^[2]
Adobe After Effects Refine Edge tool ^[4]
YUVSoft Matting plugin for Adobe After Effects ^[11]

To enhance the speed and quality of matting, some methods use additional data. For example, time-of-flight cameras had been explored in real-time matting systems.^[12]

Background replacement

Another application of video matting is background matting, which is very popular in online video calls. A Zoom plugin had been developed,^[13] and Skype announced Background Replace in June 2020.^[14] Video matting methods also allow to apply video effects only to background or foreground.

3D video editing

Video matting is crucial in 2D to 3D conversion, where the alpha matte is used to correctly process transparent objects. It is also employed in stereo to multiview conversion.

Video completion

Closely related to matting is video completion ^[15] after removal of an object in the video. While matting is used to separate the video into several layers, completion allows to fill gaps with plausible contents from the video after removing one of the layers.

References

^ ^a ^b Chuang, Yung-Yu; Agarwala, Aseem; Curless, Brian; Salesin, David H.; Szeliski, Richard (2002). "Video matting of complex scenes". ACM Transactions on Graphics. 21 (3): 243–248. doi:10.1145/566654.566572. ISSN 0730-0301.
^ ^a ^b Bai, Xue; Wang, Jue; Simons, David; Sapiro, Guillermo (2009). "Video SnapCut". ACM Transactions on Graphics. 28 (3): 1–11. doi:10.1145/1531326.1531376. ISSN 0730-0301.
^ ^a ^b Levin, A.; Lischinski, D.; Weiss, Y. (2008). "A Closed-Form Solution to Natural Image Matting". IEEE Transactions on Pattern Analysis and Machine Intelligence. 30 (2): 228–242. doi:10.1109/TPAMI.2007.1177. ISSN 0162-8828. PMID 18084055.
^ ^a ^b Bai, Xue; Wang, Jue; Simons, David (2011). "Towards Temporally-Coherent Video Matting". Computer Vision/Computer Graphics Collaboration Techniques. Lecture Notes in Computer Science. Vol. 6930. pp. 63–74. doi:10.1007/978-3-642-24136-9_6. ISBN 978-3-642-24135-2. ISSN 0302-9743.
^ Xu, Ning; Price, Brian; Cohen, Scott; Huang, Thomas (2017). "Deep Image Matting". 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 311–320. doi:10.1109/CVPR.2017.41. ISBN 978-1-5386-0457-1. S2CID 14061786.
^ ^a ^b ^c Erofeev, Mikhail; Gitman, Yury; Vatolin, Dmitriy; Fedorov, Alexey; Wang, Jue (2015). "Perceptually Motivated Benchmark for Video Matting". Proceedings of the British Machine Vision Conference 2015. pp. 99.1–99.12. doi:10.5244/C.29.99. ISBN 978-1-901725-53-7.
^ Cao, Guangying; Li, Jianwei; Chen, Xiaowu; He, Zhiqiang (2017). "Patch-based self-adaptive matting for high-resolution image and video". The Visual Computer. 35 (1): 133–147. doi:10.1007/s00371-017-1424-3. ISSN 0178-2789. S2CID 24625947.
^ Kambhamettu, Chandra (2009). "Learning based digital matting". 2009 IEEE 12th International Conference on Computer Vision. IEEE. pp. 889–896. doi:10.1109/iccv.2009.5459326. ISBN 978-1-4244-4420-5.
^ Karacan, Levent; Erdem, Aykut; Erdem, Erkut (2015). "Image Matting with KL-Divergence Based Sparse Sampling". 2015 IEEE International Conference on Computer Vision (ICCV). pp. 424–432. doi:10.1109/ICCV.2015.56. ISBN 978-1-4673-8391-2. S2CID 2174306.
^ Wang, Jue; Bhat, Pravin; Colburn, R. Alex; Agrawala, Maneesh; Cohen, Michael F. (2005). "Interactive video cutout". ACM Transactions on Graphics. 24 (3): 585–594. doi:10.1145/1073204.1073233. ISSN 0730-0301.
^ "Matting plugin for Adobe After Effects". Retrieved 2021-03-02.
^ Wang, Liang; Gong, Minglun; Zhang, Chenxi; Yang, Ruigang; Zhang, Cha; Yang, Yee-Hong (2011-06-15). "Automatic Real-Time Video Matting Using Time-of-Flight Camera and Multichannel Poisson Equations". International Journal of Computer Vision. 97 (1). Springer Science and Business Media LLC: 104–121. doi:10.1007/s11263-011-0471-x. ISSN 0920-5691. S2CID 255108880.
^ "Real-Time High Resolution Background Matting". Retrieved 2021-03-02.
^ "Introducing Background Replace in Skype". Retrieved 2021-03-02.
^ "Video Completion Benchmark". Retrieved 2021-03-10.

External links

[ChuangAgarwala2002-1] Chuang, Yung-Yu; Agarwala, Aseem; Curless, Brian; Salesin, David H.; Szeliski, Richard (2002). "Video matting of complex scenes". ACM Transactions on Graphics. 21 (3): 243–248. doi:10.1145/566654.566572. ISSN 0730-0301.

[BaiWang2009-2] Bai, Xue; Wang, Jue; Simons, David; Sapiro, Guillermo (2009). "Video SnapCut". ACM Transactions on Graphics. 28 (3): 1–11. doi:10.1145/1531326.1531376. ISSN 0730-0301.

[LevinLischinski2008-3] Levin, A.; Lischinski, D.; Weiss, Y. (2008). "A Closed-Form Solution to Natural Image Matting". IEEE Transactions on Pattern Analysis and Machine Intelligence. 30 (2): 228–242. doi:10.1109/TPAMI.2007.1177. ISSN 0162-8828. PMID 18084055.

[BaiWang2011-4] Bai, Xue; Wang, Jue; Simons, David (2011). "Towards Temporally-Coherent Video Matting". Computer Vision/Computer Graphics Collaboration Techniques. Lecture Notes in Computer Science. Vol. 6930. pp. 63–74. doi:10.1007/978-3-642-24136-9_6. ISBN 978-3-642-24135-2. ISSN 0302-9743.

[XuPrice2017-5] Xu, Ning; Price, Brian; Cohen, Scott; Huang, Thomas (2017). "Deep Image Matting". 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 311–320. doi:10.1109/CVPR.2017.41. ISBN 978-1-5386-0457-1. S2CID 14061786.

[ErofeevGitman2015-6] Erofeev, Mikhail; Gitman, Yury; Vatolin, Dmitriy; Fedorov, Alexey; Wang, Jue (2015). "Perceptually Motivated Benchmark for Video Matting". Proceedings of the British Machine Vision Conference 2015. pp. 99.1–99.12. doi:10.5244/C.29.99. ISBN 978-1-901725-53-7.

[CaoLi2017-7] Cao, Guangying; Li, Jianwei; Chen, Xiaowu; He, Zhiqiang (2017). "Patch-based self-adaptive matting for high-resolution image and video". The Visual Computer. 35 (1): 133–147. doi:10.1007/s00371-017-1424-3. ISSN 0178-2789. S2CID 24625947.

[Kambhamettu_2009_p.-8] Kambhamettu, Chandra (2009). "Learning based digital matting". 2009 IEEE 12th International Conference on Computer Vision. IEEE. pp. 889–896. doi:10.1109/iccv.2009.5459326. ISBN 978-1-4244-4420-5.

[KaracanErdem2015-9] Karacan, Levent; Erdem, Aykut; Erdem, Erkut (2015). "Image Matting with KL-Divergence Based Sparse Sampling". 2015 IEEE International Conference on Computer Vision (ICCV). pp. 424–432. doi:10.1109/ICCV.2015.56. ISBN 978-1-4673-8391-2. S2CID 2174306.

[WangBhat2005-10] Wang, Jue; Bhat, Pravin; Colburn, R. Alex; Agrawala, Maneesh; Cohen, Michael F. (2005). "Interactive video cutout". ACM Transactions on Graphics. 24 (3): 585–594. doi:10.1145/1073204.1073233. ISSN 0730-0301.

[11] "Matting plugin for Adobe After Effects". Retrieved 2021-03-02.

[Wang_Gong_Zhang_Yang_pp._104–121-12] Wang, Liang; Gong, Minglun; Zhang, Chenxi; Yang, Ruigang; Zhang, Cha; Yang, Yee-Hong (2011-06-15). "Automatic Real-Time Video Matting Using Time-of-Flight Camera and Multichannel Poisson Equations". International Journal of Computer Vision. 97 (1). Springer Science and Business Media LLC: 104–121. doi:10.1007/s11263-011-0471-x. ISSN 0920-5691. S2CID 255108880.

[13] "Real-Time High Resolution Background Matting". Retrieved 2021-03-02.

[14] "Introducing Background Replace in Skype". Retrieved 2021-03-02.

[15] "Video Completion Benchmark". Retrieved 2021-03-10.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

v t e Video processing
Post-processing	Deblocking Resizing Comparison Deinterlacing Denoising Deflicking
Special processing	Film colorization (tinting) Color grading Film look Super-resolution imaging Video matting Uncompressed Pixel art scaling Telecine