Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition

This is the official implementation of the paper "Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognition" by H.Kwon, M.Kim, S.Kwak, and M.Cho. For more information, checkout the project website and the paper on arXiv.

Environment:

• Cuda: 9.0
• gcc: 7.3.0
• Python 3.6.8
• PyTorch 1.0.1
• TorchVison: 0.2.2
• Spatial Correlation Sampler
• Others: environment.yml

Anaconda environment setting

git clone https://github.com/arunos728/SELFY.git
cd selfy
conda env create -f environment.yml
conda activate selfy

Installing Correlation sampler

cd Pytorch-Correlation-extension
python setup.py install

# check whether SpatialCorrelationSampler is installed correctly.
python check.py forward
python check.py backward
python checkCorrelationSampler.py

Please check this repo for the detailed instructions.

Dataset preparation

Please refer to TSM repo for the detailed data preparation instructions.

File lists (.txt files in ./data) specify configurations of each video clips (path, #frames, class). We upload our Something-Something-V1 & V2 video file lists in ./data. The path of the file lists should be added into the scripts for training (or testing).

Training & Testing

• For training SELFYNet on Something-Something, use the following command:

    ./scripts/train_SELFY_Something.sh

• For testing your trained model on Something-Something, use the following command:

    ./scripts/test_SELFY_Something.sh

Citation

If you use this code or ideas from the paper for your research, please cite our paper:

@inproceedings{kwon2021learning,
  title={Learning self-similarity in space and time as generalized motion for video action recognition},
  author={Kwon, Heeseung and Kim, Manjin and Kwak, Suha and Cho, Minsu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={13065--13075},
  year={2021}
}

Contact

Heeseung Kwon([email protected]), Manjin Kim([email protected])

Questions can also be left as issues in the repository. We will be happy to answer them.