Implementation of the Point Transformer layer, in Pytorch

No one can reproduce the performance reported in your original paper. Please post your pre-trained model or your original code. Otherwise, we must question your academic ethics!****

I wrote some wrapper code to turn this layer into a full transformer and I can't seem to figure out what is going wrong. The following works:

import torch
from torch import nn, einsum
import x_transformers
from point_transformer_pytorch import PointTransformerLayer

layer = PointTransformerLayer(
    dim = 7,
    pos_mlp_hidden_dim = 64,
    attn_mlp_hidden_mult = 4,
    num_neighbors = 16          # only the 16 nearest neighbors would be attended to for each point
)

feats = torch.randn(1, 5, 7)
pos = torch.randn(1, 5, 3)
mask = torch.ones(1, 5).bool()

y = layer(feats, pos, mask = mask)

However this doesn't work

import torch
from torch import nn, einsum
import x_transformers
from point_transformer_pytorch import PointTransformerLayer

class PointTransformer(nn.Module):
    def __init__(self, feats, mask, neighbors = 16, layers=5, dimension=5):
        
        super().__init__()
        
        self.feats = feats
        self.mask = mask
        self.neighbors = neighbors
        
        self.layers = []
        
        for _ in range(layers):
            self.layers.append(PointTransformerLayer(
                dim = dimension,
                pos_mlp_hidden_dim = 64,
                attn_mlp_hidden_mult = 4,
                num_neighbors = self.neighbors
            ))

    def forward(self, pos):
        curr_pos = pos
        for layer in self.layers:
            print(curr_pos)
            curr_pos = layer(self.feats, pos, self.mask)
            print("----")
        return curr_pos

model = PointTransformer(feats, mask)
model(pos)

The error I'm getting is mat1 and mat2 shapes cannot be multiplied (5x7 and 5x15)

Great job! I have a question about the number of the points in the point cloud. Do you have any suggestion to deal with point clouds with different point. As I know, point cloud models are always applied in Shapenet which contains point clouds with 2048 points. So what can we do if the number of the point clouds is not constant?

It seems that the implementation of the multi-head point transformer produces scalar attention scores for each head.

https://github.com/lucidrains/point-transformer-pytorch/blob/99bc3958138d8c9d3b882e4ac50b1a18a86160fe/point_transformer_pytorch/multihead_point_transformer_pytorch.py#L62

Hi,

Thanks for this contribution. In the implementation of attn_mlp the first linear layer increases the dimension. Is this a standard practice because I did not find any details about this in the paper. Also paper also does not describe use of mask, is this again some standard practice for attention layers?

Thanks!!

Dear Authors, In your paper you wrote: "The layer is invariant to permutation and cardinality and is thus inherently suited to point cloud processing."

I do not understand this statement, because your PointTransformerLayer https://github.com/lucidrains/point-transformer-pytorch/blob/main/point_transformer_pytorch/point_transformer_pytorch.py#L31 requires the dim parameter in initialization. So it always expects dim elements in input. What if a point cloud has dim+1 points?

Thank you in advance.

I'm not sure whether I used the point-transformer correctly: I just implemented one block for training, and the data shape of (x, pos) in each gpu are both [16, 2048, 3], later I was informed that my gpu is running out of the memory(11.77 GB total capacity)