Projective Analysis for 3D Shape Segmentation

ACM Transactions on Graphics

(Proceedings of SIGGRAPH Asia 2013)

Yunhai Wang1   Minglun Gong2   Tianhua Wang3  Daniel Cohen-Or4   Hao Zhang5   Baoquan Chen1 6   
1Shenzhen VisuCA Key Lab/SIAT     2Memorial University of Newfoundland     3Jilin University    
4Tel-Aviv University     5Simon Fraser University     6Shandong University    

     


Figure 1: Our projective analysis treats an input 3D model (a) as a collection of projections (b), which are labeled (d) based on selected images (c) from a pre-labeled image database. Back-projecting 2D labels onto the 3D model forms a probability map (e), which allows us to infer the final shape segmentation and labeling (f). Note how the labeling of the twin stroller is inferred from the images of single strollers.

Abstract

We introduce projective analysis for semantic segmentation and labeling of 3D shapes. The analysis treats an input 3D shape as a collection of 2D projections, labels each projection by transferring knowledge from existing labeled images, and back-projects and fuses the labelings on the 3D shape. The image-space analysis involves matching projected binary images of 3D objects based on a novel bi-class Hausdorff distance. The distance is topology-aware by accounting for internal holes in the 2D figures and it is applied to piecewise-linearly warped object projections to compensate for part scaling and view discrepancies. Projective analysis simplifies the processing task by working in a lower-dimensional space, circumvents the requirement of having complete and well-modeled 3D shapes, and addresses the data challenge for 3D shape analysis by leveraging the massive available image data. A large and dense labeled set ensures that the labeling of a given projected image can be inferred from closely matched labeled images. We demonstrate semantic labeling of imperfect (e.g., incomplete or self-intersecting) 3D models which would be otherwise difficult to analyze without taking the projective analysis approach.

Paper and Supplemental material

Paper (7M)
Supplemental material (8M)
Slides (M)

Results
     

Figure 2: Labeling results on various imperfect meshes downloaded from Trimble Warehouse. In each row, from left to right, we show the input shape, the top two ranked images and label transfer results, probability map, and the final labeling under two different views.
     

Data

To achive our projective shape analysis, we developed the PSA dataset. You can download all the sets of labeled 2D photos and 3D imperfect shapes we used from there.

Code

This archive contains the code of BiSH distance that we developed for our algorithm.

Acknowledgements

The authors would like to thank all the reviewers for their valuable comments. This work is supported in part by grants from NSFC (61202222, 61232011, 61025012, 61202223), 863 Program (2013AA01A604), Guangdong Science and Technology Program (2011B050200007), Shenzhen Science and Innovation Program
(CXB201104220029A, ZD201111080115A, KC2012JSJS0019A), Natural Science and Engineering Research Council of Canada (293127 and 611370) and the Israel Science Foundation.