AIST++ API This repo contains starter code for using the AIST++ dataset.

Overview

AIST++ API

This repo contains starter code for using the AIST++ dataset. To download the dataset or explore details of this dataset, please go to our dataset website.

Installation

The code has been tested on python>=3.7. You can install the dependencies and this repo by:

pip install -r requirements.txt
python setup.py install

You also need to make sure ffmpeg is installed on your machine, if you would like to visualize the annotations using this api.

How to use

We provide demo code for loading and visualizing AIST++ annotations. Note AIST++ annotations and videos, as well as the SMPL model (for SMPL visualization only) are required to run the demo code.

The directory structure of the data is expected to be:


├── motions/
├── keypoints2d/
├── keypoints3d/
├── splits/
├── cameras/
└── ignore_list.txt


└── *.mp4


├── SMPL_MALE.pkl
└── SMPL_FEMALE.pkl

Visualize 2D keypoints annotation

The command below will plot 2D keypoints onto the raw video and save it to the directory ./visualization/.

python demos/run_vis.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --video_dir <VIDEO_DIR> \
  --save_dir ./visualization/ \
  --video_name gWA_sFM_c01_d27_mWA2_ch21 \
  --mode 2D

Visualize 3D keypoints annotation

The command below will project 3D keypoints onto the raw video using camera parameters, and save it to the directory ./visualization/.

python demos/run_vis.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --video_dir <VIDEO_DIR> \
  --save_dir ./visualization/ \
  --video_name gWA_sFM_c01_d27_mWA2_ch21 \
  --mode 3D

Visualize the SMPL joints annotation

The command below will first calculate the SMPL joint locations from our motion annotations (joint rotations and root trajectories), then project them onto the raw video and plot. The result will be saved into the directory ./visualization/.

python demos/run_vis.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --video_dir <VIDEO_DIR> \ 
  --smpl_dir <SMPL_DIR> \
  --save_dir ./visualization/ \ 
  --video_name gWA_sFM_c01_d27_mWA2_ch21 \ 
  --mode SMPL

Multi-view 3D keypoints and motion reconstruction

This repo also provides code we used for constructing this dataset from the multi-view AIST Dance Video Database. The construction pipeline starts with frame-by-frame 2D keypoint detection and manual camera estimation. Then triangulation and bundle adjustment are applied to optimize the camera parameters as well as the 3D keypoints. Finally we sequentially fit the SMPL model to 3D keypoints to get a motion sequence represented using joint angles and a root trajectory. The following figure shows our pipeline overview.

AIST++ construction pipeline overview.

The annotations in AIST++ are in COCO-format for 2D & 3D keypoints, and SMPL-format for human motion annotations. It is designed to serve general research purposes. However, in some cases you might need the data in different format (e.g., Openpose / Alphapose keypoints format, or STAR human motion format). With the code we provide, it should be easy to construct your own version of AIST++, with your own keypoint detector or human model definition.

Step 1. Assume you have your own 2D keypoint detection results stored in , you can start by preprocessing the keypoints into the .pkl format that we support. The code we used at this step is as follows but you might need to modify the script run_preprocessing.py in order to be compatible with your own data.

python processing/run_preprocessing.py \
  --keypoints_dir <KEYPOINTS_DIR> \
  --save_dir <ANNOTATIONS_DIR>/keypoints2d/

Step 2. Then you can estimate the camera parameters using your 2D keypoints. This step is optional as you can still use our camera parameter estimates which are quite accurate. At this step, you will need the /cameras/mapping.txt file which stores the mapping from videos to different environment settings.

# If you would like to estimate your own camera parameters:
python processing/run_estimate_camera.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --save_dir <ANNOTATIONS_DIR>/cameras/
# Or you can skip this step by just using our camera parameter estimates.

Step 3. Next step is to perform 3D keypoints reconstruction from multi-view 2D keypoints and camera parameters. You can just run:

python processing/run_estimate_keypoints.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --save_dir <ANNOTATIONS_DIR>/keypoints3d/

Step 4. Finally we can estimate SMPL-format human motion data by fitting the 3D keypoints to the SMPL model. If you would like to use another human model such as STAR, you will need to do some modifications in the script run_estimate_smpl.py. The following command runs SMPL fitting.

python processing/run_estimate_smpl.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --smpl_dir <SMPL_DIR> \
  --save_dir <ANNOTATIONS_DIR>/motions/

Note that this step will take several days to process the entire dataset if your machine has only one GPU. In practise, we run this step on a cluster, but are only able to provide the single-threaded version.

MISC.

  • COCO-format keypoint definition:
[
"nose", 
"left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder","right_shoulder", 
"left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hip", "right_hip", 
"left_knee", "right_knee", "left_ankle", "right_ankle"
]
  • SMPL-format body joint definition:
[
"root", 
"left_hip", "left_knee", "left_foot", "left_toe", 
"right_hip", "right_knee", "right_foot", "right_toe",
"waist", "spine", "chest", "neck", "head", 
"left_in_shoulder", "left_shoulder", "left_elbow", "left_wrist",
"right_in_shoulder", "right_shoulder", "right_elbow", "right_wrist"
]
Owner
Google
Google ❤️ Open Source
Google
hey, this repo is the backend of the sociio project

sociio backend Hey, this repository is a part of sociio project , In this repo we are working to create an independent server for everything you can i

2 Jun 09, 2022
It's an .exe file that can notify your chia profit and warning message every time automatically.

chia-Notify-with-Line 警示程式 It's an .exe file that can notify your chia profit and warning message every time automatically. 這是我自行設計的小程式,有轉成.exe檔了,可以在沒

You,Yu 1 Oct 28, 2021
BDD base project: Python + Behave

BDD base project: Python + Behave Basic example of using Python with Behave (BDD). This Gherkin example includes: Basic Scenario Scenario Outline Tagg

eccanto 1 Dec 08, 2021
🙌Kart of 210+ projects based on machine learning, deep learning, computer vision, natural language processing and all. Show your support by ✨ this repository.

ML-ProjectKart 📌 Repository This kart showcases the finest collection of all projects based on machine learning, deep learning, computer vision, natu

Prathima Kadari 203 Dec 28, 2022
Pre-crisis Risk Management for Personal Finance

Антикризисный риск-менеджмент личных финансов Риск-менеджмент личных финансов условиях санкций и/или финансового кризиса: делаем сегодня все, чтобы за

Dmitry Petukhov 593 Jan 09, 2023
PythonCalculator - A simple Calculator made in python using tkinter for GUI

PythonCalculator A simple Calculator made in python using tkinter for GUI. For P

ʀᴇxɪɴᴀᴢᴏʀ 1 Jan 01, 2022
In this project we will implement AirBnB clone using console

AirBnB Clone In this project we will implement AirBnB clone using console. Usage The shell should work like this

Nandweza Allan 1 Feb 07, 2022
Custom component to calculate estimated power consumption of lights and other appliances

Custom component to calculate estimated power consumption of lights and other appliances. Provides easy configuration to get virtual power consumption sensors in Home Assistant for all your devices w

Bram Gerritsen 552 Dec 28, 2022
BlackIP-Rep is a tool designed to gather the reputation and information of Bulk IP's.

BlackIP-Rep is a tool designed to gather the reputation and information of Bulk IP's. Focused on increasing the workflow of Security Operations(SOC) team during investigation.

0LiVEr 6 Dec 12, 2022
A simple hash system.

PBH-Hash-System A simple hash system. Usage You could use it like this: from pbh import pbh print(pbh("Hey", True)) Output: 2feae2471698cfcdcbd6b98ca

Karim 3 Mar 24, 2022
PyPI package for scaffolding out code for decision tree models that can learn to find relationships between the attributes of an object.

Decision Tree Writer This package allows you to train a binary classification decision tree on a list of labeled dictionaries or class instances, and

2 Apr 23, 2022
A small Blender addon for changing an object's local orientation while in edit mode

A small Blender addon for changing an object's local orientation while in edit mode.

Jonathan Lampel 50 Jan 06, 2023
To attract customers, the hotel chain has added to its website the ability to book a room without prepayment

To attract customers, the hotel chain has added to its website the ability to book a room without prepayment. We need to predict whether the customer is going to reject the booking or not. Since in c

Taychinov Evgeniy 0 Aug 04, 2022
Code needed for hybrid land cover change analysis for NASA IDS project

Documentation for the NASA IDS change analysis Poley 10/21/2021 Required python packages: whitebox numpy rasterio rasterio.mask os glob math itertools

Andrew Poley 2 Nov 12, 2021
Mmr image postbot - Бот для создания изображений с новыми релизами в сообщество ВК MMR Aggregator

Mmr image postbot - Бот для создания изображений с новыми релизами в сообщество ВК MMR Aggregator

Max 3 Jan 07, 2022
Small C-like language compiler for the Uxn assembly language

Pyuxncle is a single-pass compiler for a small subset of C (albeit without the std library). This compiler targets Uxntal, the assembly language of the Uxn virtual computer. The output Uxntal is not

CPunch 13 Jun 28, 2022
WATTS provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level

WATTS (Workflow and Template Toolkit for Simulation) provides a set of Python classes that can manage simulation workflows for multiple codes where information is exchanged at a coarse level.

13 Dec 23, 2022
Tesla App Update Differences Extractor

Tesla App Update Differences Extractor Python program that finds the differences between two versions of the Tesla App. When Tesla updates the app a l

Adrian 5 Apr 11, 2022
Audio2Face - a project that transforms audio to blendshape weights,and drives the digital human,xiaomei,in UE project

Audio2Face - a project that transforms audio to blendshape weights,and drives the digital human,xiaomei,in UE project

FACEGOOD 732 Jan 08, 2023
Sathal's Python Projects Repository

Sathal's Python Projects Repository Purpose and Motivation I come from a mainly C Programming Language background and have previous classroom experien

Sam 1 Oct 20, 2021