This PR adds the following:

• [x] Scripts to install OpenDR toolkit in clean Ubuntu 20.04 systems (even when running from a minimal image, e.g,. docker ones)
• [x] setup.py to correctly install OpenDR package
• [x] Corrected scripts to activate OpenDR venv environment
• [x] Scripts to create bdist wheels for cpu only usage
• [x] Dockerfile for assembling cpu-only OpenDR inference
• [x] Readme listing different installation options
• [x] Update wiki to reflect the changes made in this PR

This PR also add missing __init__.py in toolkit and a __version__ variable according to typical python usage.

This PR upgrades the toolkit to using CUDA 11.2. This also ensures that the toolkit will be compatible with the NVIDIA 30xx GPUs. For PyTorch we are using precompiled packages that bundle CUDA11.1. This does not affect the system-wide CUDA version.

This PR also improves testing using GPUs, as well as fixes some documentation issues regarding the use of OPENDR_DEVICE variable.

Tasks to be performed

• [x] Change dockerfile to using CUDA 11.2
• [x] Update PyTorch and mxnet
• [x] Update detectron
• [x] Update DCNv2
• [x] Make sure that pip installation does not need any kind of update
• [x] Update the documentation if needed

We need to restore github branch in Dockerfile prior to merging.

This is a PR for synthetic multi-view facial image generator which will be a standalone tool of OPENDR generating data (facial images) for procedures such as e.g. training.

Hi everyone,

This is an initial version with our approach on mobile manipulation based on our paper (https://arxiv.org/abs/2101.05325). It's not completely ready to be merged yet, but should already include all the main parts.

• It implements the LearnerRL interface
• Formatted according to PEP-8, .clang-format
• Most unnecessary functionality should already be removed
• Includes a first version of the documentation, including examples to train and to evaluate provided checkpoints

But there are also a few questions from my side. Mainy because this is a project that relies on a python3-based interface for the user and an environment implemented in c++, which additionally draws on functionality from ROS (mainly moveit).

• Atm I am keeping the c++ src and header files within the module, combined with it's own CMakeLists.txt. (i.e. within src/control/mobile_manipulation/). Is that appropriate?
• How should I define ROS and c++ dependencies? The module provides environments for several robotic platforms (PR2, Tiago, HSR). This means to compile or run the module the user needs (i) a ROS installation (developped and tested for melodic) (ii) a separate catkin_ws for each robot (iii) launch a launchfile before running the python scripts. Due to this I feel like it makes sense to not require every user of other openDR modules to install these, but rather have them specified specific to this module. Some of robot specific dependencies should furthermore be compiled in separate catkin_workspaces. The model checkpoints are tiny (3x 3MB) and currently directly located in the git repo. Is that ok for such small files?
• Licenses: this module includes slightly modified launchfiles from openly available ROS packages (~/robots_worlds/[pr2/hsr/tiago]). Do these have to be marked or treated specially somehow?
• This was developped as part of WP5.2 Deep Navigation. As there is no navigation folder and this approach can be seen as a combination of navigation and control I have located it within control for now. Let me know in case I should move it elsewhere

Any help on the above would be much appreciated. Other comments on what is already in here are also welcome as well.

Some remaining todo's for myself to remember:

• update checkpoints
• test that gazebo evaluation works
• test that examples in readme work

This PR adds two learners (which train and evaluate a baseline model and three proposed models) for skeleton-based human action recognition.

• A new data type named SkeletonSequence is added to the engine.data.py and a new target class named ActionCategory is added to engine.target.py.

• The learners' implementation follows the provided template and sufficient tests are provided for all the functions which will be directly called by the user including, fit(), eval(), infer(), save(), load(), optimize(), multi_stream_eval(), network_builder().

This PR contains a new ROS2 (Foxy-Fitzroy version) workspace located in the projects directory and, for now, it serves to ~test and finalize the structure, naming, etc.,~ gather and finalize all ROS2 nodes in a unified PR. Right now there are no ~docstrings~ (docstrings added), documentation or READMEs. This description will get updated for any additions.

Contents:

1. opendr_perception python package
   • Contains a pose estimation node
   • Contains fall detection node
   • Contains object detection 2d centernet/detr/ssd/yolov3 nodes
   • Contains face detection retinaface node
   • Contains face recognition node
   • Contains semantic segmentation bisenet node
   • ~Contains a subscriber tester node (tester), that subscribes to the messages published by the pose estimation node for testing~ Testing can be performed as described in steps 9 and 10 of Building and Running below.
2. opendr_ros2_bridge python package
   • Contains bridge.py which includes a class with methods to convert images, poses, etc. from and to ROS2 messages
   • This uses cv_bridge which is included in the vision_opencv package
3. opendr_ros2_messages CMake package
   • Contains custom OpenDR messages, right now only OpenDRPose2D and OpenDRPose2DKeypoint
   • Note that this package is a CMake package due to limitations in ROS2: custom messages cannot be generated in python packages

The logic behind the structuring of the packages and nodes is similar to OpenDR's ROS1 packages/nodes.

Below you can find instructions to install, build and run the nodes for testing. Note that i did everything on a system with ROS1 already installed.

I faced many issues along the way that might reappear in a fresh install of ROS2, etc., so if any problems/errors occur following the instructions please get in touch with me, to possibly save you some time.

Installation

• To install ROS2 i followed this tutorial (section 2), which installs the 'foxy' release of ROS2. (Note that on '(7) configure environment variables', you need to replace dashing with foxy)
• Edit: At this point you might need to run sudo apt-get install ros-foxy-vision-msgs as discussed below
• Install colcon, basically just sudo apt install python3-colcon-common-extensions
• Install ros2 usb cam to test with local webcam. In my case i use ros2 run usb_cam usb_cam_node_exe to run it after installation, which seems to work fine

Building and Running

1. Navigate to your OpenDR installation and activate it as usual
2. Navigate to workspace root, opendr_ws_2 directory
3. Install cv_bridge via the instructions in its README, excluding the last step(build). There seems to be no need to build it, as it will get built along with the rest of the packages later.
4. Navigate to the workspace root (opendr_ws_2) as the previous step leaves you inside vision_opencv dir
5. Run colcon build
6. Run . install/setup.bash
7. Run ros2 run opendr_perception pose_estimation to start the pose estimation node (or any other existing node)
8. In a new terminal run ros2 run usb_cam usb_cam_node_exe to grab images from a webcam
9. In a new terminal run ros2 run rqt_image_view rqt_image_view and select the corresponding topic to view the image result
10. In a new terminal run ros2 topic echo opendr/poses to view the pose message. Note that it is not really human readable in that form, it should be read in another node and converted into an OpenDR pose object to have access to human-friendly print methods.

* If you are using conda, check out Illia's comment down below. Thanks @iliiliiliili !

To be added

ROS2 nodes to be added according to what ROS1 nodes exist already:

Perception package:

• [x] Object detection 2D detr (update from original author) (#296)
• [x] Video activity recognition (#323)
• [x] RGBD hand gesture recognition (#341)
• [x] Panoptic segmentation EfficientPS (#270)
• [x] Heart anomaly detection(#337)
• [x] Speech command recognition ( #340)
• [x] Audiovisual emotion recognition (#342)
• [x] Skeleton based action recognition (#344)
• [x] Landmark-based facial expression recognition (#345)
• [x] Image-based facial emotion estimation (new tool #264, #346)
• [x] Object detection 2D gem (#295)
• [x] Object detection 2D YOLOv5 (added in #360, I will directly add the ROS2 node on ros2 branch)
• [x] Object detection 2D Nanodet (added in #278, I will directly add the ROS2 node on ros2 branch)
• [x] Object tracking 2D SiamRPN (added in #367, ~I will directly add the~ WIP ROS2 node on ros2 branch)
• [x] High resolution pose estimation (added in #356, I will directly add the ROS2 node on ros2 branch)
• [x] Image dataset (#319)
• [x] Point cloud dataset (#319)
• [x] Object detection 3D voxel (#319)
• [x] Object tracking 2D deep sort (#319)
• [x] Object tracking 2D fair mot (#319)
• [x] Object tracking 3D ab3dmot (#319)

Data generation package:

• [x] Synthetic facial recognition (#288)

Simulation package:

• [x] Human model generation client/service (#291)

Planning package:

• [x] End to end planner (this is new for ROS1 too) (~~#286, new PR will be opened for ROS2~~ #358)

Edit1: Updated the last steps of the instructions as well as the contents list as per the latest changes. Edit2: Added information in contents list about the new opendr_ros2_messages. added TODO list for remaining nodes

This PR adds image-based facial expression recognition and valence-arousal estimation. It includes learner, unit-tests, demo, document, and ROS node. This is a replacement for the previous PR which had conflicts with other tools.

Hi, this PR adds the EfficientPS network. The original repo can be found here.

~~Please also check issue #90.~~

Todos:

• [x] Upload pre-trained models to OpenDR server and adjust the URLs in efficient_ps_learner.py.
• [x] Add unit tests
• [x] Add documentation to /docs/reference
• [x] Merge Heatmap implementation with version proposed in #100 ~~and updates pending in #98~~ ~~Install CUDA in GitHub CI~~ --> will not be resolvable since the code requires GPUs. See comment.

Known issues:

• ~~reason for failing tests: 3rd party dependencies assume an existing pytorch installation since they attempt to load torch in their setup.py~~

Hi All,

This is an initial version of our method on end-to-end local planning. It's not completely ready to be merged yet, but should already include all the main parts.

• It implements the LearnerRL interface
• Formatted according to PEP-8
• Includes a first version of the documentation

Remaining todo's for myself:

• Tests for code

Creating a new PR due to a forced pushed. See #68 for initial discussion. To recap the open points from the initial PR:

• the license on the tiago urdf -> contacted PAL
• the unittests -> currently blocked by the missing linting support for typing

Parameters need to be consistent with other tools with agparser

I am using opendr installed on my computer on the develop branch, I am feeding the rgb camera topic and the depth_image like this one:

I cannot get an output from the /opendr/gestures topic Is the depth_image topic different from the one that you are using?

The creation was successful, however the root package wasn't uploaded due to a missing new line in packages.txt. I've uploaded the missing one manually, this is just to ensure everything is fine

The follow PR contains:

1. More tools in C api.
2. New data structures for tensors manipulation in C.
3. Better Json parser with arrays and floats.
4. Docs.
5. Small changes in face_recognition and nanodet_jit (naming parameters as said in wiki).
6. Tests in new data structures and tools.
7. Python api bug fixes in open pose and fair mot for onnx optimizations.

As emerged here https://github.com/opendr-eu/opendr/pull/381 without the upper restriction on numpy, the version 1.24.0 might be installed and in it several deprecation warnings have expired. Even when things work, several deprecation warnings are printed when running the tests. Both issues should be addressed.

I was unable to find it documented so i am opening a new issue with the following error for the deepsort ROS1 node:

[ERROR] [1671020211.255272]: bad callback: <bound method ObjectTracking2DDeepSortNode.callback of <__main__.ObjectTracking2DDeepSortNode object at 0x7edf5c2f28>>
Traceback (most recent call last):
  File "/opt/ros/noetic/lib/python3/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/opendr/projects/opendr_ws/src/opendr_perception/scripts/object_tracking_2d_deep_sort_node.py", line 105, in callback
    tracking_boxes = self.learner.infer(image_with_detections, swap_left_top=True)
  File "/opendr/src/opendr/perception/object_tracking_2d/deep_sort/object_tracking_2d_deep_sort_learner.py", line 289, in infer
    result = self.tracker.infer(image, frame_id, swap_left_top=swap_left_top)
  File "/opendr/src/opendr/perception/object_tracking_2d/deep_sort/algorithm/deep_sort_tracker.py", line 81, in infer
    bbox_xywh[:, 3:] *= 1.2
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

What i found the node works properly when provided with images from the image_dataset_node but throws this error when taking input from a webcam.

Hi all,

This PR is to add ROS2 node for EfficientLPS. This PR should be merged after #359 is merged. It includes EfficientLPS node and PointCloud2 Publisher node in it. It will remain as a draft until #359 is merged.