PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space（2017）

1 Introduction

點雲的特點：

① collections of points in a Euclidean space

② invariant to permutations of its members

③ the distance metric defifines local neighborhoods that may exhibit different properties（距離度量定義不同屬性鄰域）

the density and other attributes of points may not be uniform across different locations — in 3D scanning the density variability can come from perspective effects, radial density variations, motion, etc.（不同位置的點的密度和其他屬性可能不一致——在3D掃描中，密度的可變性可能來自透視效果、徑向密度變化、運動等）

 

The basic idea of PointNet is to learn a spatial encoding of each point and then aggregate all individual point features to a global point cloud signature. By its design, PointNet does not capture local structure induced by the metric.

 

PointNet++（general idea）：

① We first partition the set of points into overlapping local regions by the distance metric of the underlying space.

② Similar to CNNs, we extract local features capturing fine geometric structures from small neighborhoods;

③ such local features are further grouped into larger units and processed to produce higher level features.

④ This process is repeated until we obtain the features of the whole point set.

 

two issues that need to be addressed： 

① how to generate the partitioning of the point set 

② how to abstract sets of points or local features through a local feature learner  （PointNet）

 

解決第二個問題：

Choose local feature learner to be PointNet，As a basic building block, PointNet abstracts sets of local points or features into higher level representations. In this view, PointNet++ applies PointNet recursively on a nested partitioning of the input set.

 

解決第一個問題：

Each partition is defifined as a neighborhood ball in the underlying Euclidean space, whose parameters include centroid location and scale.The centroids are selected by a farthest point sampling (FPS) algorithm.

PointNet++ leverages neighborhoods at multiple scales.

（FPS算法找質心，多尺度利用鄰域）

 

 

2 Problem Statement

：離散度量空間

：點集

d：距離度量

M的密度非均勻

輸入：，和每個點的其他屬性

學習：集合函數 f

f 可以是對X打標籤的分類函數，也可以是爲M中每個點打標籤的分割函數

 

 

3 Method

 

3.1 Review of PointNet

Given an unordered point set ，

define a set function f：  

maps a set of points to a vector：

γ and h are usually multi-layer perceptron (MLP) networks

 

 

 

3.2 Hierarchical Point Set Feature Learning

 

 

① The Sampling layer selects a set of points from input points, which defifines the centroids of local regions.

② Grouping layer then constructs local region sets by finding “neighboring” points around the centroids.

③ PointNet layer uses a mini-PointNet to encode local region patterns into feature vectors.

 

N × (d + C)

N points with d-dim coordinates and C-dim point feature

 subsampled points with d-dim coordinates and new -dim feature vectors summarizing local context.

 

Sampling layer

① Given input points 

② use iterative farthest point sampling (FPS) to choose a subset of points，

such that xij is the most distant point (in metric distance) from the set {xi1 , xi2 , ..., xij 1 } with regard to the rest points.

 

Grouping layer

input：a point set of size N × (d + C)  and the coordinates of a set of centroids of size

output：groups of point sets of size  ，each group corresponds to a local region and K is the number of points in the neighborhood of centroid points（不同組之間K可能是不同的，但是接下來的PointNet層可以變爲固定長度的區域特徵向量）

 

和KNN確定鄰域相比，球查詢的鄰域保證了固定的區域尺度

 

PointNet layer

input：

output：

 

首先進行座標轉換：，

is the coordinate of the centroid

 

 

3.3 Robust Feature Learning under Non-Uniform Sampling Density

 

hierarchical network + density adaptive PointNet layers = PointNet++

 

 

Multi-scale grouping (MSG)

We train the network to learn an optimized strategy to combine the multi-scale features. This is done by randomly dropping out input points with a randomized probability for each instance, which we call random input dropout. Specififically, for each training point set, we choose a dropout ratio θ uniformly sampled from [0, p] where p ≤ 1. For each point, we randomly drop a point with probability θ. In practice we set p = 0.95 to avoid generating empty point sets. In doing so we present the network with training sets of various sparsity (induced by θ) and varying uniformity (induced by randomness in dropout). During test, we keep all available points.(組合策略是要訓練的)

 

Multi-resolution grouping (MRG)

features of a region at some level Li is a concatenation of two vectors. One vector (left in figure) is obtained by summarizing the features at each subregion from the lower level Li-1 using the set abstraction level. The other vector (right) is the feature that is obtained by directly processing all raw points in the local region using a single PointNet.

 

MRG計算量更高效

 

3.4 Point Feature Propagation for Set Segmentation

 

（default: p=2,k=3）

 

 

4 Experiments

4.3 Point Set Classifification in Non-Euclidean Metric Space

 

4.4 Feature Visualization

 

 

---

farthest point sampling (FPS) algorithm  選擇質心

FPS的好處：Compared with random sampling, it has better coverage of the entire point set given the same number of centroids.

 

L. Luciano and A. B. Hamza. Deep learning with geodesic moments for 3d shape classifification. Pattern Recognition Letters, 2017.

L. Yi, H. Su, X. Guo, and L. Guibas. Syncspeccnn: Synchronized spectral cnn for 3d shape segmentation. arXiv preprint arXiv:1612.00606, 2016.

 

geodesic neighborhood

Euclidean neighborhood

 

擾動對象位置和點樣本位置