A python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization

#59 #58 Right now, I have a version of uncertainty_toolbox uploaded to PyPI and Anaconda.

pip install uncertainty_toolbox

conda install -c sgbaird uncertainty_toolbox

The basic instructions (after some one-time setup) are to install flit (e.g. conda install flit), update the version number in uncertainty_toolbox/__init__.py and run:

flit publish

to upload a new version to PyPI. I probably need to add other people's usernames to PyPI so I'm not the only one that can upload new versions.

For the Anaconda upload, install conda-souschef and grayskull, run a slightly customized script, and build it within the scratch folder.

conda install conda-souschef grayskull
python run_grayskull.py
cd scratch
conda build .

For this, you probably have to set a few things with conda first, such as automatic uploads when building, credentials, and configuring it to look in certain channels. These are all one-time setup instructions. I also have some GitHub workflow code in mat_discover that can take care of the uploads (and testing) automatically when you make a new release. Just need to change a couple lines and add credentials to GitHub secrets.

New added UTs consider:

• function calls with no arguments for both assert_is_flat_same_shape() and assert_is_positive()

• new UTs for assert_is_positive() function

• based on assert_is_positive() description, a UT with 2D ndarray was included too.

@willieneis @YoungseogChung @IanChar @HanGuo97

Test execution output:

Hi,

Thanks for making this package available to us! I have a simple question:

I have a data set split into train/validation sets. A regressor (or classifier) is trained, and then by using the validation set I get an estimate Y_prediction. From the same validation set, I have the Y_true. So, how would you suggest to compute the standard deviation on predictions from a regressor (or classifier)?

Many thanks,

Ivan

Hello I have tried to install the package in google colab according to your instruction. It dose not work. would you help me?

Thanks for the great package

Hi, I have a trained neural network. How can I used the Uncertainty Toolbox to quantify uncertainty of the neural network. How should I calculate predictions_std (a vector of standard deviations)?

New added UTs consider:

• function calls with no arguments for both assert_is_flat_same_shape() and assert_is_positive()
• new UTs for assert_is_positive() function
• based on assert_is_positive() description, a UT with 2D ndarray was included too.

@willieneis @YoungseogChung @IanChar @HanGuo97

Test execution output:

Incorporating some code into another repository, and wanted to check with some people with (very likely) a more thorough statistics background than myself.

Conversion from standard deviation to confidence intervals takes place in: https://github.com/uncertainty-toolbox/uncertainty-toolbox/blob/b2f342f6606d1d667bf9583919a663adf8643efe/uncertainty_toolbox/metrics_scoring_rule.py#L187

What is the inverse conversion from symmetric confidence intervals back to standard deviation? (e.g. using scipy.stats.norm but doesn't have to be)

I'm trying to figure out how to describe the math behind interval score in a simple way. Based on reading the paper and related literature, I get that it incorporates width and coverage. Is the idea that it penalizes the following two scenarios:

• wide intervals
• when many predictions fall outside of the intervals

In other words, the interval should be as narrow as possible while having the interval "cross the parity line" as frequently as possible, correct? How does this play out in the math?

Hello, thanks for sharing your code!

I noticed that you implement the metric "MACE", while I was more familiar with the term "ECE". Is there a difference between the two?

Hello!

Thanks for this convenient tool! I was wondering how you best suggest using it for quantile regression. I see that a lot of functions take in a pred-mean and std. However, I thought it would also be nice if there were additional support for quantiles and not only std + mean support. Is there a way of hacking my way through what is now so that I could use the quantile intervals I get instead?

When dealing with "large" datasets of over 100 data points, scatter plots can become cluttered. Additionally, it's difficult to really distinguish points that are on top of each other. So a cluster of 1000 data points could look the same as a cluster of 100, but they mean very different things.

Seaborn's bivariate displots can get around this issue by showing a density of points rather than each individual point. It'd be nice to have the option to make such plots for, say, plot_xy or plot_residuals_vs_stdevs.

I acknowledge this would be a messy request, but it'd be a feature I'd use.

Some requirements are not needed in the regular requirements right now.

1. Shapely is no longer needed. The imports in several files first need to be removed, and plot_calibration (in viz.py) needs to use the new calibration calculation code.
2. Black and py-test can be added to the dev requirements instead of regular requirements.

@tailintalent found that many of the evaluation metrics can be greatly sped up with gpu support via pytorch. This code is currently in an experimental branch called "torch". We will keep it separate from the main package for now. In order to be merged into main we will need the following:

• The code relying on torch needs to be separated out because we do not want to make torch a mandatory install. This needs to be done in the most robust way possible, ideally separating by pinpointing the exact operations where torch provides a speedup rather than making a torch version of the pre-existing code.
• Set up an optional installation that allows users to select the torch-enabled version of the toolbox.
• Add options to the functions that allows users to select to use the torch version of the metric rather than the numpy version.

I've had at least two different folks come to me being very confused about the shapes of calibration curves. After a few minutes of discussion, it turns out the confusion arose from how we plot the observed values are on the y-axis and predicted values are on the x-axis. This is the opposite of conventional parity plots, where observations are on the x-axis and predictions are on the y-axis.

What do folks thing about flipping the axes? I know this would be different than the original Kuleshov paper's graphs (and also the paper that @willieneis and I worked on together), but a part of me would rather plot what folks would expect to minimize future confusion.

After running:

pip install git+https://github.com/uncertainty-toolbox/uncertainty-toolbox

on Windows 10, VS Code, inside of conda environment, I get:

Could not find lib geos_c.dll or load any of its variants [].
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\shapely\geos.py", line 54, in load_dll
    raise OSError(
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\shapely\geos.py", line 178, in <module>
    _lgeos = load_dll("geos_c.dll")
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\shapely\coords.py", line 10, in <module>
    from shapely.geos import lgeos
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\shapely\geometry\base.py", line 20, in <module>
    from shapely.coords import CoordinateSequence
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\shapely\geometry\__init__.py", line 4, in <module>
    from .base import CAP_STYLE, JOIN_STYLE
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\uncertainty_toolbox\metrics_calibration.py", line 10, in <module>
    from shapely.geometry import Polygon, LineString
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\uncertainty_toolbox\metrics.py", line 9, in <module>
    from uncertainty_toolbox.metrics_calibration import (
  File "C:\Users\sterg\miniconda3\envs\vickers-hardness\Lib\site-packages\uncertainty_toolbox\__init__.py", line 9, in <module>
    from .metrics import (
  File "C:\Users\sterg\Documents\GitHub\sparks-baird\VickersHardnessPrediction\hv_prediction.py", line 21, in <module>
    import uncertainty_toolbox as uct

By installing shapely from conda (https://stackoverflow.com/questions/56813083/oserror-could-not-find-geos-c-dll-or-load-any-of-its-variants),

conda install shapely

For some reason (not sure if this would be the same for everyone), I need to do an uninstall and reinstall it one more time:

conda uninstall shapely
conda install shapely

and it seems to work OK now.