A Temporal Extension Library for PyTorch Geometric

I'm wondering if there is an example of classification carried out for data represented as temporal graphs?

To be precise: in my use case I have for instance a sequence of 100 graphs representing together a single entity/object. Next, I have 200 entities. Is there an easy way to input those 200 data instances (with ground truth classification labels) to one of the numerous deep learning architectures implemented in this repo to obtain instance representations useful in the classification task? Or simply obtain predicted labels? Is there an example for this or did I miss it ? (apologies if I did).

Thank you for your help in advance.

Hi, I've been trying to reuse the example code, MPNNLSTM, GCLSTM, and GConvLSTM with my own dataset. My MSE errors are around .2 but the models do not return correct target values. When I run the examples the MSE is around 1. What does MSE represent? How should I structure data my data for these models? I have a graph with 20 nodes, 2 features per node, and 32 edges. The dataset has 360 separate graphs. My dataset looks similar to the one used in MPNNLSTM.

Hi @benedekrozemberczki, I found that the gradient function may be lost in the implementation of EvolveGCN. The reason for this should be the following line of code: https://github.com/benedekrozemberczki/pytorch_geometric_temporal/blob/bd6fac946378507a729b5723e9ace46a9dc0cbe1/torch_geometric_temporal/nn/recurrent/evolvegcno.py#L69 And here is a demo try to verify it:

import torch

torch.manual_seed(123)


class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.toy_model = ToyModel(weight=True)

    def forward(self, x):
        W = self.toy_model.weight
        self.toy_model.weight = torch.nn.parameter.Parameter(W)
        return self.toy_model(x)


class MyModelParam(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.W = torch.nn.parameter.Parameter(torch.Tensor(2, 3).fill_(0.5))
        self.toy_model = ToyModel(weight=False)

    def forward(self, x):
        W = self.W
        return self.toy_model(x, weight=W)


class ToyModel(torch.nn.Module):
    def __init__(self, weight=True):
        super().__init__()
        if weight:
            self.weight = torch.nn.parameter.Parameter(torch.Tensor(2, 3).fill_(0.5))
        else:
            self.register_parameter('weight', None)
        # self.func = torch.nn.parameter.Parameter(torch.Tensor(2, 2).fill_(0.1))

    def forward(self, x, weight=None):
        if weight is not None:
            w = weight
        else:
            w = self.weight
        # x = self.func * x
        x = torch.matmul(x, w)
        return x


x = torch.Tensor(1, 2).fill_(0.8)
y = torch.Tensor(1, 3).fill_(0.7)
print("################## Test 1: reset GCN.weight by nn.Parameter()##################")
model = MyModel()
print("toyModel weight before:")
print(model.toy_model.weight)
optim = torch.optim.SGD(model.parameters(), lr=1e-2)

# print("x: {}".format(x))
# print("y: {}".format(y))
prediction = model(x)
loss = (prediction - y).sum()
loss.backward()
optim.step()
print("toyModel weight after:")
print(model.toy_model.weight)

########################## Test 2
print("################## Test 2: pass weight as a parameter during forward##################")
model_param = MyModelParam()
print("model param weight before:")
print(model_param.W)
optim_param = torch.optim.SGD(model_param.parameters(), lr=1e-2)
x_param = torch.Tensor(1, 2).fill_(0.8)
y_param = torch.Tensor(1, 3).fill_(0.7)
# print("x_param: {}".format(x_param))
# print("y_param: {}".format(y_param))
prediction_param = model_param(x_param)
loss_param = (prediction_param - y_param).sum()
loss_param.backward()
optim_param.step()
print("model param weight after:")
print(model_param.W)

result is here:

################## Test 1: reset GCN.weight by nn.Parameter()##################
toyModel weight before:
Parameter containing:
tensor([[0.5000, 0.5000, 0.5000],
        [0.5000, 0.5000, 0.5000]], requires_grad=True)
toyModel weight after:
Parameter containing:
tensor([[0.5000, 0.5000, 0.5000],
        [0.5000, 0.5000, 0.5000]], requires_grad=True)
################## Test 2: pass weight as a parameter during forward##################
model param weight before:
Parameter containing:
tensor([[0.5000, 0.5000, 0.5000],
        [0.5000, 0.5000, 0.5000]], requires_grad=True)
model param weight after:
Parameter containing:
tensor([[0.4920, 0.4920, 0.4920],
        [0.4920, 0.4920, 0.4920]], requires_grad=True)

Process finished with exit code 0

We can see that if we set model.weight in each batch iter by nn.paramter.Parameter(), the weight has not been updated.
One way to fix it should be to pass a new weight during every forward. But pyG does not seem to provide the corresponding interface.

Hi Benedek!

I wonder if there is a tutorial for creating a custom dataset for multiple dynamic graphs?

I want to follow a code snippet so I can play around to learn how to do that.

Thanks in advance. :)

This includes:

• a new data-structure StaticHeteroGraphTemporalSignal that works like StaticGraphTemporalSignal but with torch_geometric HeteroData objects as snapshots; instead of np arrays, dictionaries wirth key(node/edge types as strings)/value(indices/features as np arrays) are expected
• code tests for the new data-structure
• documentation for the new data-structure

Feedback is welcome! If you think it makes sense to extend your package like that it would be great to open a new branch for heterogeneous support where we can go on working in this direction. Thanks and cheers, Gregor

Hi Benedek! First of all, thank you for this project, I hope it has a bright future ahead. My issue is with building deep/multi-layer models out of Recurrent Graph Convolutional Layers. There are no examples of it in the repo (or anywhere else on the internet as far as I searched), so I might be doing something wrong. Here is my project. Now, what I observe when running this code is that RAM utilization goes up nearly exponentially with the number of RGC layers. Of course, most of it ends up in swap, making the training process too slow to be viable. This does not appear to be a memory leak, since most of the memory is not used by Python objects, but rather internally by PyTorch. Have you encountered this issue and is there a way to fix it?

I am trying to distribute the model computation in multiple GPUs using DataParallel from Pytorch Geometric library I was trying to follow this example but I am running into errors. Is this the way to do it or should I look somewhere else? Are there any examples out there to distribute models from Pytorch Geometric Temporal library?

Hi,

I've created a Dataset object which currently returns a single sequence (StaticGraphTemporalSignal) from a directory of many sequences. When feeding this to a DataLoader, I receive lists of length batch_size instead of batched objects. Is there a straightforward procedure of combining multiple StaticGraphTemporalSignal objects into a StaticGraphTemporalSignalBatch object? I have not found any examples of StaticGraphTemporalSignalBatch in use.

Cheers.

I'm dealing with a basketball players trajectory forecasting dataset; it is both temporal and spatial with the shape: [number of plays, number of timesteps, player, position]. Each play is composed of several timesteps, so in total I would have number of plays * number of timesteps graphs.

From reading documentation and source code it seems that the StaticGraphTemporalSignal accepts multiple graphs, but I'm not sure how to do that. Moreover, I cannot create an instance of this iterator without passing target labels, which is not what I aim for in this project; unless I pass the target labels as the features of the next step in the prediction.

I'm not sure if my question is clear I would be glad to elaborate, I require help :)

Hi all, just thought I would let you know that not all your tests pass when running on a machine with a GPU

I was looking at the MTGNN test which started working once I set the device to CPU and wasn't working with the existing code

   #device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    device = torch.device("cpu")

=========================================================================== short test summary info ============================================================================ FAILED test/convolutional_test.py::test_astgcn - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! FAILED test/convolutional_test.py::test_mstgcn - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! FAILED test/convolutional_test.py::test_gman - RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward =================================================================== 3 failed, 33 passed in 67.57s (0:01:07) ==================================================================== (pytorch_latest_p36) sh-4.2$

Is there a plan to include Graph-Structure-Learning graph neural network models for temporal graphs? A few models with github repos include MTGNN(https://github.com/nnzhan/MTGNN) and Graph-WaveNet (https://github.com/nnzhan/Graph-WaveNet)

Hi torch geometric temporal :-)

First, thanks for this awesome lib ! Secondly, I am working on person re-identification problem and hoped to use the temporal GNN's in order to tackle the problem. As in mtm_1, I am using MediaPipe in order to extract pose estimation of the persons in different datasets (I want to contribute this datasets to this repo once I will finish my Master's). Im am trying to do a graph classification, an to train in a batch using one of the two: AAGCN/TemporalGNN. I am working with batches that means that my data is as follow:

1. node features: a tensor with a dimension of (batch, temporal dimension, number of nodes, node features).
2. ground truth: a tensor containing the ground truth person id's with a dimension of: (batch, 1).
3. edge index: since it is a static graph, this is a tensor of dimension: (2, num of connections).
4. edge attribute: tensor with all ones vector, dim: (1, number of edges).

I am trying to train a classifier with the following logic:

1. passing a batch throw temporal GNN.
2. do a global_mean_polling on the nodes dimension in order to extract graph representation.
3. passing the embeddings through linear layer for n_class classification.
4. use a loss on the prediction + embeddings.

I am not able to get a good results and wanted to ask a few question.

1. These models are able to be trained in batches?
2. I am using the Data object from torch geometric in order to load the data, is it fit to temporal signal? (I mean, until now, I treated the Data object only as a data structure, and I didn't think it ha any affect on training).
3. There is maybe an example of usage of the mtm_1 dataset?
4. Am I working right with torch geometric temporal lib? since I am working with multiple graphs (for each individual), I hoped maybe yo can give me an example of how to work with multiple graphs.

Thanks in advance for any answer/help, I hope to contribute my work at the end. I'll be able to share any further information/code upon request.

Thanks!

I believe there is a bug in the following line.

https://github.com/benedekrozemberczki/pytorch_geometric_temporal/blob/e43e95d726174574eeae32ad8c4670631fd0a58d/torch_geometric_temporal/nn/recurrent/evolvegcnh.py#L98

The evolved W that comes from the GRU unit has the same shape as X which is (nodes, features). In the GConv layer, we are passing W and performing a multiplication [email protected] but we should be performing [email protected] for the shapes to match. It happens that here W is square (due to TopKPooling) which is why it might have been missed. Therefore, I believe we need to pass W.squeeze(dim=0).T. Same goes for the O-version of EvolveGCN.

Could someone verify this please.

Thank you.

This is a great geometric temporal framework. But there are issues when I read the source code of A3TGCN and A3TGCN2. I read the source code from this repository (based on PyTorch) and compared it with the original paper and original code (based on TensorFlow). Then I found that the attention modules were different in detail. For the original version, the context vector Ct was calculated using the MLP and the hidden states of T-GCN, and should be determined as follows: and the original source code: https://github.com/lehaifeng/T-GCN/blob/e21fb99a5c56fdf667a4ff7a9c8236761f7377f8/A3T-GCN/A3T-GCN.py#L90-L109

But, this implementation just used the parameters self._attention to represent , as follows: https://github.com/benedekrozemberczki/pytorch_geometric_temporal/blob/e43e95d726174574eeae32ad8c4670631fd0a58d/torch_geometric_temporal/nn/recurrent/attentiontemporalgcn.py#L49 https://github.com/benedekrozemberczki/pytorch_geometric_temporal/blob/e43e95d726174574eeae32ad8c4670631fd0a58d/torch_geometric_temporal/nn/recurrent/attentiontemporalgcn.py#L74 https://github.com/benedekrozemberczki/pytorch_geometric_temporal/blob/e43e95d726174574eeae32ad8c4670631fd0a58d/torch_geometric_temporal/nn/recurrent/attentiontemporalgcn.py#L76-L78

Obviously, this is a modified version of A3TGCN, and the attention module has no relationship with the hidden state of the T-GCN module. In my opinion, this modification is significant, and should be marked in docs and related source code, or should be modified to be consistent with the original paper.

Thanks sincerely for your dedication! Best!

Each time the parameter is passed into the Recurrent GCN network, the value of hidden state H is always 0, and the time series characteristics of the previous time are not reflected in the second time series

Hello I have a model where I start from a single node - image and I want to progressively add nodes and edges, and in the end, evaluate the graph using the custom loss function. The features size will get progressively smaller and the node number bigger. I suppose in principle it should be possible in your framework, however, the decision to create a new node and a new connection is binary hence non-continuous hence nondifferentiable in nature how had you managed to enable gradient propagation in these conditions?

Thank you for your answer!

When running the test, all tests passed except for

test/attention_test.py:715:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py:1102: in _call_impl                                                                                                             
    return forward_call(*input, **kwargs)
torch_geometric_temporal/nn/attention/tsagcn.py:339: in forward
    y = self.relu(self.tcn1(self.gcn1(x)) + self.residual(x))
../../../anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py:1102: in _call_impl                                                                                                             
    return forward_call(*input, **kwargs)
torch_geometric_temporal/nn/attention/tsagcn.py:262: in forward                                                                                                                                                        
    y = self._non_adaptive_forward(x, y)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = UnitGCN(                                                                                                                                                                                                      (conv_d): ModuleList(
    (0): Conv2d(100, 10, kernel_size=(1, 1), stride=(1, 1))                                                                                                                                                            (1): Conv2d(100, 10, ...ack_running_stats=True)
  (soft): Softmax(dim=-2)
  (tan): Tanh()
  (sigmoid): Sigmoid()
  (relu): ReLU(inplace=True)
)
x = tensor([[[[ 0.3476,  0.1290,  0.4463,  ...,  0.4613,  0.2014,  0.0761],                                                                                                                                                  [ 1.3476,  1.1290,  1.4463,  ...,  1...,  2.4257,  2.1628],                                                                                                                                                        [ 3.6332,  4.1489,  4.0730,  ...,  4.4859,  3.4257,  3.1628]]]],                                                                                                                                                device='cuda:0')
y = None

    def _non_adaptive_forward(self, x, y):
        N, C, T, V = x.size()
        for i in range(self.num_subset):
            A1 = self.A[i]
            A2 = x.view(N, C * T, V)
>           z = self.conv_d[i](torch.matmul(A2, A1).view(N, C, T, V))
E           RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument mat2 in method wrapper_mm)

torch_geometric_temporal/nn/attention/tsagcn.py:251: RuntimeError

Similar to #46 , if I force it on CPU then it will pass

   #device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    device = torch.device("cpu")