An Agnostic Computer Vision Framework - Pluggable to any Training Library: Fastai, Pytorch-Lightning with more to come

Option to pickle records so we don't have to parse all data everytime

This option should be transparent to the user, we can expose it by a optional argument passed to DataParser.

Always a good discussion is where to store this data. Do it store it relative to the current file? Into /tmp? Or into a .mantisshrimp folder in the home directory?

Storing relative to the current file is always annoying when using version control, we have to explicitly not add it to checkout

Example

COCOParser(data, source, use_cached=True)

• I have added the code to support most torchvision bacbones (except alexnet and Densenet)

I have unit tested the mantisshrimp/backbones/torchvision_backbones.py file.

How do I test the entire fasterRCNN code I am unsure. Please let me know so that I can test and create PRs next time.

🚀 Feature

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

Hi Aditya,

RandomSplitter is not able to split the train and validation dataset properly from WheatParser. Could you please correct the format ( def filepath(self, o) -> Union[str, Path]: return self.source / f"{o.image_id}.jpg" ) code from WheatParser class from https://www.kaggle.com/okeaditya/fastaiv2-pytorch-lightning-fasterrcnn

🐛 Bug

The prediction contains a reference to the image array. If any stage of the workflow starts accumulating the predictions objects (as it currently happens in COCOMetric this will result in a memory leak.

COCOMetric currently uses a "quickfix" to make sure the images are not being stored (take a look at accumulate), but we need a more general solution.

📓 New <Tutorial/Example>

Request for an example

What is the task? Object detection using transfer learning for the whole architecture. Are there some defined methods to load fastai model and change its head to a different number of classes, similar to this?

I was able to run the Faster-RCNN example using this example trained on COCO dataset and evaluate its mAP.

The EfficientDet workflow seems not to be yet ready. Has there been some update on that?

I was able to create EfficientDet with pretrained encoder and train it myself on COCO. I'm now trying to do transfer learning for a different number of classes. Loading model through fastai, expectedly, throws an error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-cf184f01fec3> in <module>
----> 1 learn.load('coco_local')

~/anaconda3/envs/icevision/lib/python3.8/site-packages/fastai/learner.py in load(self, file, with_opt, device, **kwargs)
    293         if self.opt is None: self.create_opt()
    294         file = join_path_file(file, self.path/self.model_dir, ext='.pth')
--> 295         load_model(file, self.model, self.opt, device=device, **kwargs)
    296         return self
    297 

~/anaconda3/envs/icevision/lib/python3.8/site-packages/fastai/learner.py in load_model(file, model, opt, with_opt, device, strict)
     47     hasopt = set(state)=={'model', 'opt'}
     48     model_state = state['model'] if hasopt else state
---> 49     get_model(model).load_state_dict(model_state, strict=strict)
     50     if hasopt and with_opt:
     51         try: opt.load_state_dict(state['opt'])

~/anaconda3/envs/icevision/lib/python3.8/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1049 
   1050         if len(error_msgs) > 0:
-> 1051             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
   1052                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1053         return _IncompatibleKeys(missing_keys, unexpected_keys)

RuntimeError: Error(s) in loading state_dict for DetBenchTrain:
	size mismatch for model.class_net.predict.conv_pw.weight: copying a param with shape torch.Size([819, 64, 1, 1]) from checkpoint, the shape in current model is torch.Size([99, 64, 1, 1]).
	size mismatch for model.class_net.predict.conv_pw.bias: copying a param with shape torch.Size([819]) from checkpoint, the shape in current model is torch.Size([99]).

Is this example for a specific model? EfficientDet

Is this example for a specific dataset? COCO transfer learning

Don't remove Main issue for examples: #39

In order to do this, I had to remove VIABBoxParser's ability to parse bounding boxes from both rect and polygon shape attributes and assume that rect annotations are for bounding boxes and polygon annotations for masks. If not, then the polygon annotations were added as an additional bbox and this didn't allow it to be parsed due to length mismatch betwen number of boxes and masks.

I've left some of the older code commented out rather than delete it, will be happy to clean that up after some feedback.

• With the current change I added CI for more OS systems. It is good idea to build on multiple OS before publishing.

• This technique is taken from PyTorch Lightning.

• I have added CI for releasing to both PyPi test server and PyPi official. This is again taken from Lightning

• We need a test to build package as well. And test PyPi releasing with a dummy release.

🚀 Feature

Hub came into a lot of reformulations, but it's real essence is becoming clear, a place to share datasets (and more)!

For a specific dataset, the following functions can be present:

• Load dataset (obligatory)
• Implement parser (obligatory)
• Implement baseline model (optional)
• Implement metrics (optional)
• Additional stuff we cannot predict, for example, the user may want to include some visualization functions, some exploratory data-analysis tool, etc... (optional)

A great place to draw inspiration is huggingface/nlp, where they maintain a collection of datasets.

While they maintain a very easy to use interface, I think there are some points we can improve.

huggingface/nlp locks the interface and provide fixed methods for getting the dataset, e.g. load_dataset. This works very well for simple (predicted) use cases, but means that custom (unpredicted) functionality is very hard (or impossible) to implement.

Instead of implementing the interface ourselves, it's a better idea to leave the implementation to the contributor, the only thing we need to do is provide are "blueprints" for the obligatory and common optional methods.

For example, in huggingface for getting the pets dataset we would do:

dataset = load_dataset('pets') #additional arguments can be passed

What I'm proposing would be:

dataset = hub.pets.load_dataset() #additional arguments can be passed

All datasets on hub would have at least the obligatory functions, plus any optional functions. Discoverability of custom functions would be easy because of autocompletion: hub.pets.<TAB> would show everything available.

What is really cool about huggingface/nlp is that they figured out a way of automatically testing the users datasets, they automatically generate a sample of data from the contributor dataset and test on that. Awesome.

@ai-fast-track, @oke-aditya, what do you think?

🚀 Feature

If targets are available, we could also show how the predictions compare to the targets.

Maybe a side by side view? Or maybe a difference view between targets and predictions.

Maybe something like this (from this article):

🚀 Feature

We need a quick demo on how to use streamlit with mantisshrimp. The first step in to get it running locally.

We should also add helpers functions to make the process easier.

Some COCO Datasets already have encoded RLEs so I added support to the parser to handle cases where they are already encoded. I also removed the use of iscrowd as a method of determining wether it is RLE or Polygon data as in my opinion that is not the proper use of that tag in the dictionary as the mask storage method doesn't always correlate with iscrowd.

Here is the issue that I brought up. https://github.com/airctic/icevision/issues/1165#issue-1513291363

🚀 Feature

Current problem I have a COCO Dataset that is storing segmentation data in the following format.

"annotations": [
      {
         "segmentation": {
            "size": [
               512,
               512
            ],
            "counts": "X][email protected][email protected]>d0N1O1O2N1N2O1O1N2N1N3M2M3lA^Nh=d1UBbNf=a1XBbNd=`1[BaNd=`1[BaNd=`1[BbNc=n1O1O1O1O1O1O001O1O1O1O10O01O001O1O001O1O1O^MmBY2R=;O1O10XMPC`2o<9O1O1O1O101N1O100O10O0100O01O1N2M2O2O1O100O100O010O1O10O0100O1O1H8K5N2O1N3O101PNWBe1k=WN_BQ1GXO[>JhA4L05NY>JmA1J33MT?1nSe1"
         },
         "area": 11312,
         "iscrowd": 1,
         "image_id": 2,
         "bbox": [
            304,
            351,
            101,
            112
         ],
         "keypoints": [],
         "num_keypoints": 0,
         "category_id": 2,
         "id": 0
      }, ...

It is clear from this that the count value is being stored as an encoded RLE rather than a normal RLE, as ice vision seems to be assuming.

Best Solution I think the best solution to this problem is to add a check in the COCODatasetParser that determines whether the data being passed in is in the encoded RLE format or just normal RLE format. I understand right now that iscrowd is used to determine whether it is RLE information or just polygon information. I also think this approach is slightly hacky because plenty of COCODatasets arent necessarily storing crowd segmentation data but still use RLE due to the data size advantages. It would be great if this could be detected automatically which I think is pretty reasonable.

For Polygons you can check wether segmentation is a dict of information or just an array. If it is just an array then that means it is stored as polygon data.

For RLE data you can check if it is a dict and if the count's value contains a list of ints then you know it is standard RLE.

For Encoded RLE Data you can check if the information in counts is a String and if it is then you can just immediately provide the Encoded RLE.

Alternative Solutions

1. I can most likely modify this before going into icevision by decoding the dataset and using just normal RLEs but that seems like a roundabout way.
2. I can also change to simply using polygons rather than RLE. Both of these solutions require modifying my existing dataset rather than just adding support.

🚀 Feature

Is your feature request related to a problem? Please describe. Yes, I can't take advantage of MPS

Describe the solution you'd like Support newer versions of pytorch

Describe alternatives you've considered I've tried to install pytorch 1.13 but seems that 'nms' isn't available in torchvision.ops.boxes

The previous Installation Using Conda Guide link in CONTRIBUTING.md did not take the user to the required page

The link "https://airctic.com/install/#b-installation-using-conda" was replaced with "https://airctic.com/0.12.0/install/#installation-using-conda" and takes us to the proper site in the docs

🐛 Bug

Hi there, when I import icevision in colab I get an error: ModuleNotFoundError: No module named 'sahi.model'

To Reproduce Steps to reproduce the behavior: !pip install icevision[all] from icevision.all import *

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-5-2dc2b878494d>](https://localhost:8080/#) in <module>
----> 1 from icevision.all import *

2 frames
[/usr/local/lib/python3.7/dist-packages/icevision/__init__.py](https://localhost:8080/#) in <module>
      5 from icevision.data import *
      6 from icevision import backbones
----> 7 from icevision import models
      8 from icevision.metrics import *
      9 from icevision.visualize import *

[/usr/local/lib/python3.7/dist-packages/icevision/models/__init__.py](https://localhost:8080/#) in <module>
     30 
     31 if SoftDependencies.sahi:
---> 32     from icevision.models import inference_sahi

[/usr/local/lib/python3.7/dist-packages/icevision/models/inference_sahi.py](https://localhost:8080/#) in <module>
     12 
     13 
---> 14 from sahi.model import DetectionModel
     15 from sahi.prediction import ObjectPrediction
     16 from sahi.predict import get_sliced_prediction as sahi_get_sliced_prediction

ModuleNotFoundError: No module named 'sahi.model'

Screenshots

Where it happens:

• colab (possibly others, but I didn't have a chance to check)