This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild with Dense 3D Representations and A Benchmark. (CVPR 2022)"

Related tags

Deep Learningpythonpytorchgaitgait-recognitiongait-in-the-wildgait3dsmplgait
Overview

Gait3D-Benchmark

This is the code for the paper "Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei: Gait Recognition in the Wild with Dense 3D Representations and A Benchmark. (CVPR 2022)". The official project page is here.

What's New

  • [Mar 2022] Another gait in the wild dataset GREW is supported.
  • [Mar 2022] Our Gait3D dataset and SMPLGait method are released.

Model Zoo

Gait3D

Input Size: 128x88(64x44)

Method [email protected] [email protected] mAP mINP download
GaitSet(AAAI2019)) 42.60(36.70) 63.10(58.30) 33.69(30.01) 19.69(17.30) model-128(model-64)
GaitPart(CVPR2020) 29.90(28.20) 50.60(47.60) 23.34(21.58) 13.15(12.36) model-128(model-64)
GLN(ECCV2020) 42.20(31.40) 64.50(52.90) 33.14(24.74) 19.56(13.58) model-128(model-64)
GaitGL(ICCV2021) 23.50(29.70) 38.50(48.50) 16.40(22.29) 9.20(13.26) model-128(model-64)
OpenGait Baseline* 47.70(42.90) 67.20(63.90) 37.62(35.19) 22.24(20.83) model-128(model-64)
SMPLGait(CVPR2022) 53.20(46.30) 71.00(64.50) 42.43(37.16) 25.97(22.23) model-128(model-64)

*It should be noticed that OpenGait Baseline is equal to SMPLGait w/o 3D in our paper.

Cross Domain

Datasets in the Wild (GaitSet, 64x44)

Source Target [email protected] [email protected] mAP
GREW (official split) Gait3D 15.80 30.20 11.83
GREW (our split) 16.50 31.10 11.71
Gait3D GREW (official split) 18.81 32.25 ~
GREW (our split) 43.86 60.89 28.06

Requirements

  • pytorch >= 1.6
  • torchvision
  • pyyaml
  • tensorboard
  • opencv-python
  • tqdm
  • py7zr
  • tabulate
  • termcolor

Installation

You can replace the second command from the bottom to install pytorch based on your CUDA version.

git clone https://github.com/Gait3D/Gait3D-Benchmark.git
cd Gait3D-Benchmark
conda create --name py37torch160 python=3.7
conda activate py37torch160
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.2 -c pytorch
pip install tqdm pyyaml tensorboard opencv-python tqdm py7zr tabulate termcolor

Data Preparation

Please download the Gait3D dataset by signing an agreement. We ask for your information only to make sure the dataset is used for non-commercial purposes. We will not give it to any third party or publish it publicly anywhere.

Data Pretreatment

Run the following command to preprocess the Gait3D dataset.

python misc/pretreatment.py --input_path 'Gait3D/2D_Silhouettes' --output_path 'Gait3D-sils-64-44-pkl' --img_h 64 --img_w 44
python misc/pretreatment.py --input_path 'Gait3D/2D_Silhouettes' --output_path 'Gait3D-sils-128-88-pkl' --img_h 128 --img_w 88
python misc/pretreatment_smpl.py --input_path 'Gait3D/3D_SMPLs' --output_path 'Gait3D-smpls-pkl'

Data Structrue

After the pretreatment, the data structure under the directory should like this

├── Gait3D-sils-64-44-pkl
│  ├── 0000
│     ├── camid0_videoid2
│        ├── seq0
│           └──seq0.pkl
├── Gait3D-sils-128-88-pkl
│  ├── 0000
│     ├── camid0_videoid2
│        ├── seq0
│           └──seq0.pkl
├── Gait3D-smpls-pkl
│  ├── 0000
│     ├── camid0_videoid2
│        ├── seq0
│           └──seq0.pkl

Train

Run the following command:

sh train.sh

Test

Run the following command:

sh test.sh

Citation

Please cite this paper in your publications if it helps your research:

@inproceedings{zheng2022gait3d,
  title={Gait Recognition in the Wild with Dense 3D Representations and A Benchmark},
  author={Jinkai Zheng, Xinchen Liu, Wu Liu, Lingxiao He, Chenggang Yan, Tao Mei},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2022}
}

Acknowledgement

Here are some great resources we benefit:

  • The codebase is based on OpenGait.
  • The 3D SMPL data is obtained by ROMP.
  • The 2D Silhouette data is obtained by HRNet-segmentation.
  • The 2D pose data is obtained by HRNet.
  • The ReID featrue used to make Gait3D is obtained by FastReID.
Comments
  • lib/modeling/models/smplgait.py throwing error when training a new dataset

    lib/modeling/models/smplgait.py throwing error when training a new dataset

    Hi Jinkai,

    When I try to use the SMPLGait to apply on other dataset, during the training process, the smplgait.py throws the error that: smpls = ipts[1][0] # [n, s, d] IndexError: list index out of range It is also interesting that I used 4 GPUs in the training. 3 of them could detect the the ipts[1][0] tensor with size 1. However, the fourth one failed to do so. Could I know how I can solve this?

    opened by zhiyuann 7
  • I have a few questions about Gait3D-Benchmark Datasets

    I have a few questions about Gait3D-Benchmark Datasets

    Hi. Im jjun. I read your paper impressively.

    We don't currently live in China, so it is difficult to use dataset on baidu disk.

    If you don't mind, is there a way to download the dataset to another disk (e.g Google drive)?

    opened by jjunnii 6
  • Question about 3D SMPL skeleton topology diagram

    Question about 3D SMPL skeleton topology diagram

    Your work promotes the application of gait recognition in real scenes, can you provide the topology diagram of the SMPL 3D skeleton in Gait3D? Because the specific meaning of the 24 joint points is not stated in your data description document.

    opened by HL-HYX 4
  • ROMP SMPL transfer

    ROMP SMPL transfer

    When I try to use the ROMP to generate out the 3D mesh, I detect there is a version conflict with the ROMP used by SMPLGait. Could I know which version of the ROMP the SMPLGait used? In this way I could use the SMPLGait to run on other ReID dataset.

    opened by zhiyuann 3
  • question about iteration and epoch

    question about iteration and epoch

    Hi! The total iteration in your code is set to 180000, and you report the total epoch as 1200 in your paper. What's the relationship between iteration and epoch?

    opened by yan811 2
  • About data generation

    About data generation

    Hi! I 'd like to know some details about data generation in NPZ files.

    In npz file: 1 What's the order of "pose"? SMPL pose parameter should be [24,3] dim, how did you convert it to [72,]? The order is [keypoint1_angel1, keypoint1_angle2, keypoint1_angle3, keypoint2_angel1, keypoint2_angle2, keypoint2_angle3...] or [keypoint1_angle1, keypoint2_angle1... keypoint1_angle2, keypoint2_angle2... keypoint1_angle3, keypoint1_angle3... ] ?

    2 How did you generate pose into SMPL format,SPIN format , and OpenPose format? What's the order of the second dim? Is the keypoint order the same with SMPL model?

    3 In pkl file: For example, dim of data in './0000/camid0_videoid2/seq0/seq0.pkl' is [48,85]. What's the order of dim 1? Is it ordered by time order or shuffled?

    opened by yan811 2
  • GREW pretreatment `to_pickle` has size 0

    GREW pretreatment `to_pickle` has size 0

    I'm trying to run GREW pretreatment code but it generates no GREW-pkl folder at the end of the process. I debugged myself and checked if the --dataset flag is set properly and the to_pickle list size before saving the pickle file. The flag is well set but the size of the list is always 0.

    I downloaded the GREW dataset from the link you guys sent me and made de GREW-rearranged folder using the code provided. I'll keep investigating what is causing such an error and if I find I'll set a fixing PR.

    opened by gosiqueira 1
  • About the pose data

    About the pose data

    Can you make a detailed description of the pose data? This is the path of one frame pose and the corresponding content of the txt file Gait3D/2D_Poses/0000/camid9_videoid2/seq0/human_crop_f17279.txt '311,438,89.201164,62.87694,0.57074964,89.201164,54.322254,0.47146344,84.92382,62.87694,0.63443935,42.150383....' I have 3 questions. Q1: what does 'f17279' means? Q2: what does the first number (e.g. 311) in the txt file mean? Q3: which number('f17279' or '311') should I regard as a base when I order the sequence? Thank you very much!

    opened by HiAleeYang 0
Owner
Official repo for Gait3D dataset
GitHub Repository
Lama-cleaner: Image inpainting tool powered by LaMa

Lama-cleaner: Image inpainting tool powered by LaMa

Qing 5.8k Jan 05, 2023
Here we present the implementation in TensorFlow of our work about liver lesion segmentation accepted in the Machine Learning 4 Health Workshop

Detection-aided liver lesion segmentation Here we present the implementation in TensorFlow of our work about liver lesion segmentation accepted in the

Image Processing Group - BarcelonaTECH - UPC 96 Oct 26, 2022
Deep Learning Package based on TensorFlow

White-Box-Layer is a Python module for deep learning built on top of TensorFlow and is distributed under the MIT license. The project was started in M

YeongHyeon Park 7 Dec 27, 2021
StrongSORT: Make DeepSORT Great Again

StrongSORT StrongSORT: Make DeepSORT Great Again StrongSORT: Make DeepSORT Great Again Yunhao Du, Yang Song, Bo Yang, Yanyun Zhao arxiv 2202.13514 Abs

369 Jan 04, 2023
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

Xin Liu 106 Dec 30, 2022
Code-free deep segmentation for computational pathology

NoCodeSeg: Deep segmentation made easy! This is the official repository for the manuscript "Code-free development and deployment of deep segmentation

André Pedersen 26 Nov 23, 2022
🧮 Matrix Factorization for Collaborative Filtering is just Solving an Adjoint Latent Dirichlet Allocation Model after All

Accompanying source code to the paper "Matrix Factorization for Collaborative Filtering is just Solving an Adjoint Latent Dirichlet Allocation Model A

Florian Wilhelm 39 Dec 03, 2022
Automated Attendance Project Using Face Recognition

dependencies for project: cmake 3.22.1 dlib 19.22.1 face-recognition 1.3.0 openc

Rohail Taha 1 Jan 09, 2022
Marine debris detection with commercial satellite imagery and deep learning.

Marine debris detection with commercial satellite imagery and deep learning. Floating marine debris is a global pollution problem which threatens mari

Inter Agency Implementation and Advanced Concepts 56 Dec 16, 2022
Implementation of SSMF: Shifting Seasonal Matrix Factorization

SSMF Implementation of SSMF: Shifting Seasonal Matrix Factorization, Koki Kawabata, Siddharth Bhatia, Rui Liu, Mohit Wadhwa, Bryan Hooi. NeurIPS, 2021

Koki Kawabata 9 Jun 10, 2022
Tensorflow implementation of "Learning Deconvolution Network for Semantic Segmentation"

Tensorflow implementation of Learning Deconvolution Network for Semantic Segmentation. Install Instructions Works with tensorflow 1.11.0 and uses the

Fabian Bormann 224 Apr 15, 2022
Merlion: A Machine Learning Framework for Time Series Intelligence

Merlion: A Machine Learning Library for Time Series Table of Contents Introduction Installation Documentation Getting Started Anomaly Detection Foreca

Salesforce 2.8k Dec 30, 2022
PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis

Daft-Exprt - PyTorch Implementation PyTorch Implementation of Daft-Exprt: Robust Prosody Transfer Across Speakers for Expressive Speech Synthesis The

Keon Lee 47 Dec 18, 2022
Differentiable rasterization applied to 3D model simplification tasks

nvdiffmodeling Differentiable rasterization applied to 3D model simplification tasks, as described in the paper: Appearance-Driven Automatic 3D Model

NVIDIA Research Projects 336 Dec 30, 2022
[EMNLP 2021] Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

RoSTER The source code used for Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training, p

Yu Meng 60 Dec 30, 2022
RetinaNet-PyTorch - A RetinaNet Pytorch Implementation on remote sensing images and has the similar mAP result with RetinaNet in MMdetection

🚀 RetinaNet Horizontal Detector Based PyTorch This is a horizontal detector Ret

Fang Zhonghao 13 Nov 19, 2022
RefineGNN - Iterative refinement graph neural network for antibody sequence-structure co-design (RefineGNN)

Iterative refinement graph neural network for antibody sequence-structure co-des

Wengong Jin 83 Dec 31, 2022
This code is part of the reproducibility package for the SANER 2022 paper "Generating Clarifying Questions for Query Refinement in Source Code Search".

Clarifying Questions for Query Refinement in Source Code Search This code is part of the reproducibility package for the SANER 2022 paper "Generating

Zachary Eberhart 0 Dec 04, 2021
This initial strategy was developed specifically for larger pools and is based on taking a moving average and deriving Bollinger Bands to create a projected active liquidity range.

Gamma's Strategy One This initial strategy was developed specifically for larger pools and is based on taking a moving average and deriving Bollinger

Gamma Strategies 46 Dec 02, 2022
Implementation of Hire-MLP: Vision MLP via Hierarchical Rearrangement and An Image Patch is a Wave: Phase-Aware Vision MLP.

Hire-Wave-MLP.pytorch Implementation of Hire-MLP: Vision MLP via Hierarchical Rearrangement and An Image Patch is a Wave: Phase-Aware Vision MLP Resul

Nevermore 29 Oct 28, 2022