Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation"

Overview

PWC

EgoNet

Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation". This repo includes an implementation that performs vehicle orientation estimation on the KITTI dataset from a single RGB image.

News:

(2021-06-21): v-0.9 (beta version) is released. The inference utility is here! For Q&A, go to discussions. If you believe there is a technical problem, submit to issues.

(2021-06-16): This repo is under final code cleaning and documentation preparation. Stay tuned and come back in a week!

Check our 5-min video (Youtube, 爱奇艺) for an introduction.

Run a demo with a one-line command!

Check instructions here.

Performance: APBEV@R40 on KITTI val set for Car (monocular RGB)

Method Reference Easy Moderate Hard
M3D-RPN ICCV 2019 20.85 15.62 11.88
MonoDIS ICCV 2019 18.45 12.58 10.66
MonoPair CVPR 2020 24.12 18.17 15.76
D4LCN CVPR 2020 31.53 22.58 17.87
Kinematic3D ECCV 2020 27.83 19.72 15.10
GrooMeD-NMS CVPR 2021 27.38 19.75 15.92
MonoDLE CVPR 2021 24.97 19.33 17.01
Ours CVPR 2021 33.60 25.38 22.80

Performance: [email protected]40 on KITTI test set for Car (RGB)

Method Reference Configuration Easy Moderate Hard
M3D-RPN ICCV 2019 Monocular 88.38 82.81 67.08
DSGN CVPR 2020 Stereo 95.42 86.03 78.27
Disp-RCNN CVPR 2020 Stereo 93.02 81.70 67.16
MonoPair CVPR 2020 Monocular 91.65 86.11 76.45
D4LCN CVPR 2020 Monocular 90.01 82.08 63.98
Kinematic3D ECCV 2020 Monocular 58.33 45.50 34.81
MonoDLE CVPR 2021 Monocular 93.46 90.23 80.11
Ours CVPR 2021 Monocular 96.11 91.23 80.96

Inference/Deployment

Check instructions here to reproduce the above quantitative results.

Training

Check instructions here to train Ego-Net and learn how to prepare your own training dataset other than KITTI.

Citation

Please star this repository and cite the following paper in your publications if it helps your research:

@InProceedings{Li_2021_CVPR,
author    = {Li, Shichao and Yan, Zengqiang and Li, Hongyang and Cheng, Kwang-Ting},
title     = {Exploring intermediate representation for monocular vehicle pose estimation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month     = {June},
year      = {2021},
pages     = {1873-1883}
}

License

This repository can be used freely for non-commercial purposes. Contact me if you are interested in a commercial license.

Links

Link to the paper: Exploring intermediate representation for monocular vehicle pose estimation

Link to the presentation video: Youtube, 爱奇艺

Relevant ECCV 2020 work: GSNet

Comments
  • Inference on custom dataset does not give proper 3d bboxes

    Inference on custom dataset does not give proper 3d bboxes

    Hi! I am trying to use the EgoNet on a custom dataset (such as Waymo Dataset) and generate 3D bboxes from the 2D predictions generated by 2D bbox detectors such as Faster R-CNN.

    A sample of KITTI predictions where 2D bboxes are output but 3D bboxes are not yet output are as follows:

    Car -1 -1 0.0 158.0 386.0 220.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
    Car -1 -1 0.0 565.0 386.0 720.0 438.0 0.0 0.0 0.0 -1.0 -1.0 -1.0 0.0 0.95
    

    However, when I set these 2D predictions as additional input for predicting bboxes using EgoNet, the 3D bboxes are not properly output at all and are just a bunch of zeros and minus-ones.

    What should I do?

    opened by joonjeon 8
  • Stream from camera

    Stream from camera

    hello, thanks for you amazing work.

    I would like to ask, what kind of changes do I need to apply in order to perform a stream from camera and detect a pose of object e.g pedestrians. Do I need to re-train your model as well?

    Thanks for answer in an advance

    opened by adamanov 7
  • Running the model on a single image

    Running the model on a single image

    Hey Nicholasli1995,

    Thanks for putting together this awesome repo, I really appreciate how thoroughly documented the setup is.

    Would it be possible to get a few hints as to how to run the model on a single image as opposed to a directory?

    I see that the data_loader in 'inference' is nearly directly from PyTorch, so it's probably my lack of experience with PyTorch that has me confused.

    Separately, I also tried making my own directory and referencing it in my .yaml file, the following command still gave me an error about referencing the KITTI dataset

    python .\inference.py --cfg "../configs/single_inference_test.yml" --visualize True --batch_to_show 1

    image

    I was a little bit surprised to see "training" in the file not found path as I'm not asking to train -- I would expect to have to place an analogous 'test.txt' in this directory with the single test image filename, but I don't think that's currently the issue.

    Thanks! Clayton

    opened by claytonkanderson 5
  • About the license for this model

    About the license for this model

    Thank you for sharing your great code. :smiley_cat:

    What is the license for this model? I'd like to cite it to the repository I'm working on if possible, but I want to post the license correctly. https://github.com/PINTO0309/PINTO_model_zoo

    Thank you.

    opened by PINTO0309 2
  • Reproduce results on the test split

    Reproduce results on the test split

    Thanks for your great work. However, I have some confusion in “Reproduce results on the test split”. If I understand correctly, the inputs to the model are 2D bounding boxes which located in “../resources/test_boxes” and the outputs should be 3D bounding boxes which placed in “../output/submission/data”. However, the results I got were not 3D bounding boxes, as follows: Drawing The results do not contain 3D information. Is it because I made a mistake somewhere?

    opened by Vncois-ZXJ 2
  • No arg_max in KITTI_train_ICRs.yml file

    No arg_max in KITTI_train_ICRs.yml file

    Hello, firstly I would like to thanks for an amazing work and that you made it public with great documentation in order to reproduce and re train the network.

    Today I wanted to train your network according to training - stage2. However, by running the code I got an error as following

    $ python train_IGRs.py --cfg "../configs/KITTI_train_IGRs.yml"
    
    => init weights from normal distribution
    => loading pretrained model ../resources/start_point.pth
    
    Total Parameters: 63,978,471
    ----------------------------------------------------------------------------------------------------------------------------------
    Total Multiply Adds (For Convolution and Linear Layers only): 19.573845863342285 GFLOPs
    ----------------------------------------------------------------------------------------------------------------------------------
    Number of Layers
    Conv2d : 306 layers   BatchNorm2d : 304 layers   ReLU : 269 layers   Bottleneck : 4 layers   BasicBlock : 108 layers   Upsample : 28 layers   HighResolutionModule : 8 layers   Sigmoid : 1 layers   
    Initializing KITTI train set, please wait...
    Found prepared keypoints at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_train_['Car'].npy
    Found prepared instance_ids at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_train_['Car']_ids.npy
    Found prepared rotations at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_train_['Car']_rots.npy
    Initialization finished for KITTI train set
    Initializing KITTI valid set, please wait...
    Found prepared keypoints at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_valid_['Car'].npy
    Found prepared instance_ids at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_valid_['Car']_ids.npy
    Found prepared rotations at ../kitti_dataset/training/keypoints/bbox9[0.332, 0.667]_valid_['Car']_rots.npy
    Initialization finished for KITTI valid set
    Traceback (most recent call last):
      File "train_IGRs.py", line 159, in <module>
        main()
      File "train_IGRs.py", line 154, in main
        train(model, model_settings, GPUs, cfgs, logger, final_output_dir)
      File "train_IGRs.py", line 89, in train
        trainer.train(train_dataset=train_dataset, 
      File "../libs/trainer/trainer.py", line 154, in train
        evaluator = Evaluator(cfgs['training_settings']['eval_metrics'], 
      File "../libs/metric/criterions.py", line 546, in __init__
        self.metrics.append(eval(metric + '(cfgs=cfgs, num_joints=num_joints)'))
      File "<string>", line 1, in <module>
      File "../libs/metric/criterions.py", line 183, in __init__
        self.arg_max = cfgs['testing_settings']['arg_max']
    KeyError: 'arg_max
    

    It seems that arg_max key is missing in the KITTI_train_IGRs.yml

    How to arg_max parameter should be defined?

    Thanks for answer in advance.

    opened by adamanov 1
  • Reproduce Result in Kitti dataset

    Reproduce Result in Kitti dataset

    Hello @Nicholasli1995 thank you for your implementation. I have several questions.

    1. What split (train and validation split) did you use for this code? is it subcnn split?
    2. during the inference using testing set do you use ROI from the object detection algorithm?
    3. Is it still use the calibration or this system can automatically generate it?
    4. How to produce the similar table as in paper? I see that you use AOS, AP, also something like Easy, Medium, Hard. However when I tried to use validation set evaluation and use kitti_eval_offline I only got some plots, and statistic informations in txt and final result as here.
    python inference.py --cfg "../configs/try_KITTI_inference:test_submission.yml"
    
    Wrote prediction file at ../result/gt_box_test/data/007025.txt
    Warning: 007025.png not included in detected images!
    PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
    ==> 1 page written on `car_detection.pdf'.
    PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
    ==> 1 page written on `car_orientation.pdf'.
    PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
    ==> 1 page written on `car_detection_ground.pdf'.
    PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
    ==> 1 page written on `car_detection_3d.pdf'.
    Thank you for participating in our evaluation!
    Loading detections...
    number of files for evaluation: 1634
      done.
    save ../result/submission/plot/car_detection.txt
    car_detection AP: 93.934082 84.951851 67.594185
    save ../result/submission/plot/car_orientation.txt
    car_orientation AP: 93.664360 84.683617 67.141930
    save ../result/submission/plot/car_detection_ground.txt
    car_detection_ground AP: 41.456226 31.880611 24.876181
    save ../result/submission/plot/car_detection_3d.txt
    car_detection_3d AP: 29.933027 24.017235 18.650723
    Your evaluation results are available at:
    ../result/submission
    

    It only produce AP no AOS. Is it because you use testing set then evaluate directly on KITTI evaluation systems so they will provide you the detail result? Suppose we want to measure in Validation set, which value must I use? because there are many information such as car_detection AP car_orientation AP car_detection_ground AP car_detection_3d AP. So in specific how to produce table 1 as in paper if we use validation set?

    -Thank you-

    opened by ftlong6666 1
  • Visualazition problem

    Visualazition problem

    Thank you for share your huge work ! Can the output coordinates of the model be directly visualized, does any transformation is needed to display them in the original image?

    opened by xiaozhihao0 0
  • Relation between kpts_3d_pred and pose

    Relation between kpts_3d_pred and pose

    Hello, Thank you for open-sourcing this amazing project!

    I have a question about the convention for the transformation of the 3D box. EgoNet only produces an egocentric pose (i.e. camera coordinates) corresponding to the rotation between the 3D box extracted from the keypoints and a template 3D box. We also have a translation corresponding to the first point in kpts_3D_pred, here.

    To better understand the coordinate systems involved I'm doing the following experiment:

    1. Create a template 3D bounding box following this, in the canonical pose.
    2. Rotate it with the rotation matrix given by EgoNet, this one

    After doing these two steps, I still need one translation to place the 3D box in space (in the camera system). The question is, what translation should I use? Is it the one corresponding to the first point in kpts_3d_pred?

    Thank you for your time

    opened by nviolante25 2
  • Generate 3D rectangular coordinates using 2D rectangular boxes

    Generate 3D rectangular coordinates using 2D rectangular boxes

    Thank you for the work you have done, and open source the code. I would like to ask how to use your code, will only 2D annotation box pictures with labels, generate 3D coordinate box used for detection tasks? I have a problem now, Is it necessary to have this file? I tried to modify it and got an error. I want to know what it does {{{ Download the resources folder and unzip its contents. Place the resource folder at ${EgoNet_DIR}/resources }}} English is translated by machine, If there is any improper wording, please forgive me Thanks you

    opened by yuan960426 4
  • Use EgoNet on custom data

    Use EgoNet on custom data

    Thank you for sharing your work. I want to use EgoNet for 3D bbox and object orientation estimation tasks on custom data. How should I proceed, do I need any other model's 2D/3D bbox predictions to start with or if I change the input data while testing I can get both 3D bbox and orientation predictions?

    opened by ratnam18 5
Releases(v-1.0)
Owner
Shichao Li
A PhD candidate @ HKUST working on computer vision and machine learning
Shichao Li
Codebase for "Revisiting spatio-temporal layouts for compositional action recognition" (Oral at BMVC 2021).

Revisiting spatio-temporal layouts for compositional action recognition Codebase for "Revisiting spatio-temporal layouts for compositional action reco

Gorjan 20 Dec 15, 2022
Deep Structured Instance Graph for Distilling Object Detectors (ICCV 2021)

DSIG Deep Structured Instance Graph for Distilling Object Detectors Authors: Yixin Chen, Pengguang Chen, Shu Liu, Liwei Wang, Jiaya Jia. [pdf] [slide]

DV Lab 31 Nov 17, 2022
Supplementary code for the AISTATS 2021 paper "Matern Gaussian Processes on Graphs".

Matern Gaussian Processes on Graphs This repo provides an extension for gpflow with Matérn kernels, inducing variables and trainable models implemente

41 Dec 17, 2022
A python script to lookup Passport Index Dataset

visa-cli A python script to lookup Passport Index Dataset Installation pip install visa-cli Usage usage: visa-cli [-h] [-d DESTINATION_COUNTRY] [-f]

rand-net 16 Oct 18, 2022
Cave Generation using metaballs in Blender. Originally created by sdfgeoff, Edited by Myself (Archie Jaskowicz).

Blender-Cave-Generation Cave Generation using metaballs in Blender. Originally created by sdfgeoff, Edited by Myself (Archie Jaskowicz). Installation

2 Dec 28, 2022
Simple, but essential Bayesian optimization package

BayesO: A Bayesian optimization framework in Python Simple, but essential Bayesian optimization package. http://bayeso.org Online documentation Instal

Jungtaek Kim 74 Dec 05, 2022
PyTorch Implementation of Region Similarity Representation Learning (ReSim)

ReSim This repository provides the PyTorch implementation of Region Similarity Representation Learning (ReSim) described in this paper: @Article{xiao2

Tete Xiao 74 Jan 03, 2023
This is the official Pytorch implementation of the paper "Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model"

Diverse Motion Stylization (Official) This is the official Pytorch implementation of this paper. Diverse Motion Stylization for Multiple Style Domains

Soomin Park 28 Dec 16, 2022
[CVPR 2021] MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition

MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition (CVPR 2021) arXiv Prerequisite PyTorch = 1.2.0 Python3 torchvision PIL argpar

51 Nov 11, 2022
Sharpness-Aware Minimization for Efficiently Improving Generalization

Sharpness-Aware-Minimization-TensorFlow This repository provides a minimal implementation of sharpness-aware minimization (SAM) (Sharpness-Aware Minim

Sayak Paul 54 Dec 08, 2022
A free, multiplatform SDK for real-time facial motion capture using blendshapes, and rigid head pose in 3D space from any RGB camera, photo, or video.

mocap4face by Facemoji mocap4face by Facemoji is a free, multiplatform SDK for real-time facial motion capture based on Facial Action Coding System or

Facemoji 591 Dec 27, 2022
High-fidelity 3D Model Compression based on Key Spheres

High-fidelity 3D Model Compression based on Key Spheres This repository contains the implementation of the paper: Yuanzhan Li, Yuqi Liu, Yujie Lu, Siy

5 Oct 11, 2022
An end-to-end project on customer segmentation

End-to-end Customer Segmentation Project Note: This project is in progress. Tools Used in This Project Prefect: Orchestrate workflows hydra: Manage co

Ocelot Consulting 8 Oct 06, 2022
Tool for installing and updating MiSTer cores and other files

MiSTer Downloader This tool installs and updates all the cores and other extra files for your MiSTer. It also updates the menu core, the MiSTer firmwa

72 Dec 24, 2022
VR Viewport Pose Model for Quantifying and Exploiting Frame Correlations

This repository contains the introduction to the collected VRViewportPose dataset and the code for the IEEE INFOCOM 2022 paper: "VR Viewport Pose Model for Quantifying and Exploiting Frame Correlatio

0 Aug 10, 2022
PyTorch implementation of "ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context" (INTERSPEECH 2020)

ContextNet ContextNet has CNN-RNN-transducer architecture and features a fully convolutional encoder that incorporates global context information into

Sangchun Ha 24 Nov 24, 2022
交互式标注软件,暂定名 iann

iann 交互式标注软件,暂定名iann。 安装 按照官网介绍安装paddle。 安装其他依赖 pip install -r requirements.txt 运行 git clone https://github.com/PaddleCV-SIG/iann/ cd iann python iann

294 Dec 30, 2022
Fast convergence of detr with spatially modulated co-attention

Fast convergence of detr with spatially modulated co-attention Usage There are no extra compiled components in SMCA DETR and package dependencies are

peng gao 135 Dec 07, 2022
pyspark🍒🥭 is delicious,just eat it!😋😋

如何用10天吃掉pyspark? 🔥 🔥 《10天吃掉那只pyspark》 🚀

lyhue1991 578 Dec 30, 2022
Towards End-to-end Video-based Eye Tracking

Towards End-to-end Video-based Eye Tracking The code accompanying our ECCV 2020 publication and dataset, EVE. Authors: Seonwook Park, Emre Aksan, Xuco

Seonwook Park 76 Dec 12, 2022