PU-GCN: Point Cloud Upsampling via Graph Convolutional Network
CVPR'2021
Abstract
The effectiveness of learning-based point cloud upsampling pipelines heavily relies on the upsampling modules and feature extractors used therein. For the point upsampling module, we propose a novel model called NodeShuffle, which uses a Graph Convolutional Network (GCN) to better encode local point information from point neighborhoods. NodeShuffle is versatile and can be incorporated into any point cloud upsampling pipeline. Extensive experiments show how NodeShuffle consistently improves state-of-the-art upsampling methods. For feature extraction, we also propose a new multi-scale point feature extractor, called Inception DenseGCN. By aggregating features at multiple scales, this feature extractor enables further performance gain in the final upsampled point clouds. We combine Inception DenseGCN with NodeShuffle into a new point upsampling pipeline called PU-GCN. PU-GCN sets new state-of-art performance with much fewer parameters and more efficient inference. Our code is publicly available at https://github.com/guochengqian/PU-GCN.
Figure 1: Effectiveness of proposed NodeShuffle and Inception DenseGCN. We propose a Graph Convolutional Network (GCN) based upsampling module NodeShuffle and a multi-scale feature extractor Inception DenseGCN. Integrating NodeShuffle into the 3PU upsampling pipeline allows for better upsampling and better structure preservation capability. We propose PU-GCN that combines both Inception DenseGCN and NodeShuffle (NS) upsampling modules. In PU-GCN, Inception DenseGCN can further improve upsampling quality and generate fine-grained details (e.g. the neck and ball shape of the faucet). The original 3PU uses duplicate-based upsampling.
Figure 2. Our proposed upsampling module: NodeShuffle. We expand the number of input features by r times using a GCN layer, and then apply a shuffle operation to rearrange the feature map. r denotes the upsampling ratio.
Figure 3. Our proposed Inception DenseGCN. We use the parameters (k, d, c) to define a DenseGCN block. k is the number of neighbors (kernel size), d is the dilation rate, and c is the number of output channels. KNN is applied at the first layer to build the graph and the node neighborhoods. Note that the d is different in the different DenseGCN blocks to achieve the goal of multi-scale feature extraction.
Figure 4. Our proposed PU-GCN rchictecture. PU-GCN uses an inception feature extractor consisting of one or more Inception DenseGCN blocks, followed by the NodeShuffle based upsampler, and a coordinate reconstructor.
PU1K Dataset
Samples from PU1K. The first and second slides contain training samples, and the third slide contains testing samples.
PU1K consists of 1,147 3D models split into 1020 training samples and 127 testing samples. The training set contains 120 3D models compiled from PU-GAN’s dataset [16], in addition to 900 different models collected from ShapeNetCore [2]. The testing set contains 27 models from PU-GAN and 100 more models from ShapeNetCore. The models from ShapeNetCore are chosen from 50 different categories. We randomly choose 200 models from each category to obtain 1,000 different models with various shape complexities to encourage diversity. Overall, PU1K covers a large semantic range of 3D objects and includes simple, as well as complex shapes.
Experiments
Figure 5. Qualitative upsampling (x4) results. We show the ×4 upsampled results of input point clouds (2048 points) when processed by different upsampling methods (PU-Net, 3PU, PU-GAN, and our PU-GCN). PU-GCN produces the best results overall, while generating fewer outliers and preserving fine-grained local details (refer to close-ups).
Figure 6. Upsampling real-scanned point clouds from KITTI dataset. PU-GCN preserves intricate structures and generates fine-grained details (e.g. the window of the car, the pedal of the motorcycle, etc.). Please zoom in to see details.
Ablation Study
Table 3:Ablation study on Inception DenseGCN. Inception DenseGCN performs better than a DenseGCN. Using two DenseGCN blocks in Inception DenseGCN to extract multi-scale information is better than using only one block. As expected, using DenseGCN in Inception DenseGCN works better than using GCN (Inception GCN). Using dilated graph convolution instead of regular graph convolution with different kernel sizes also achieves better performance with faster inference. Residual connections and global pooling inside Inception DenseGCN further improve performance. Increasing the number of Inception DenseGCN blocks tends to improve PU-GCN performance.
Table 4:Ablation study on NodeShuffle. Results show that NodeShuffle can transfer well to different upsampling architectures in the literature. Replacing the original up-sampling module with NodeShuffle improves upsampling performance overall. The effectiveness of the GCN layer inNodeShuffle is shown when replacing the GCN layer with a set of MLPs (MLPShuffle)