JumpCut: Non-Successive Mask Transfer and Interpolation for Video Cutout

SIGGRAPH Asia 2015

Qingnan Fan1    Fan Zhong1    Dani Lischinski2    Daniel Cohen-Or3     Baoquan Chen1

1Shandong University   2The Hebrew University of Jerusalem   3Tel Aviv University

Given a video frame (t - k) with a pre-segmented foreground subject, our new mask transfer approach is able to predict the mask at non-successive unsegmented frames (e.g., t + k), in a more accurate manner than by sequential frame-to-frame propagation. We are then able to accurately estimate the mask at intermediate frames, such as t, using bi-directional mask transfer, referred to as mask interpolation. Note that no manual corrections were applied to the segmentations shown in frames t + k and t. Please view the companion video!



We introduce JumpCut, a new mask transfer and interpolation method for interactive video cutout. Given a source frame for which a foreground mask is already available, we compute an estimate of the foreground mask at another, typically non-successive, target frame. Observing that the background and foreground regions typically exhibit different motions, we leverage these differences by computing two separate nearest-neighbor fields (split-NNF) from the target to the source frame. These NNFs are then used to jointly predict a coherent labeling of the pixels in the target frame. The same split-NNF is also used to aid a novel edge classifier in detecting silhouette edges (S-edges) that separate the foreground from the background. A modified level set method is then applied to produce a clean mask, based on the pixel labels and the S-edges computed by the previous two steps. The resulting mask transfer method may also be used for coherently interpolating the foreground masks between two distant source frames. Our results demonstrate that the proposed method is significantly more accurate than the existing state-of-the-art on a wide variety of video sequences. Thus, it reduces the required amount of user effort, and provides a basis for an effective interactive video object cutout tool.



Supplementary Material

More visual comparisons with SnapCut, Zhong et al. 2012 and SeamSEG
Demo of our interactive real-time video cutout system
More discriptions are in readme.pdf



Codes are also uploaded into this Github page. The video is uploaded into this Youtube page.



We thank the anonymous reviewers for their valuable comments. This paper is supported by 973 program of China (No. 2015CB352501), NSF of China (No. 61232011,61572290), Young Scholars Program of Shandong University, and by the Israel Science Foundation (ISF).



Title = {JumpCut: Non-Successive Mask Transfer and Interpolation for Video Cutout},
Author = {Qingnan Fan and Fan Zhong and Dani Lischinski and Daniel Cohen-Or and Baoquan Chen},
Journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2015)},
Year = {2015},
Volume = {34}
Number = {6},