Reusable constraint types to use with typing.Annotated

Overview

annotated-types

CI pypi versions license

PEP-593 added typing.Annotated as a way of adding context-specific metadata to existing types, and specifies that Annotated[T, x] should be treated as T by any tool or library without special logic for x.

This package provides metadata objects which can be used to represent common constraints such as upper and lower bounds on scalar values and collection sizes, a Predicate marker for runtime checks, and non-normative descriptions of how we intend these metadata to be interpreted. In some cases, we also note alternative representations which do not require this package.

Install

pip install annotated-types

Examples

from typing import Annotated
from annotated_types import Gt, Len

class MyClass:
    age: Annotated[int, Gt(18)]                         # Valid: 19, 20, ...
                                                        # Invalid: 17, 18, "19", 19.0, ...
    factors: list[Annotated[int, Predicate(is_prime)]]  # Valid: 2, 3, 5, 7, 11, ...
                                                        # Invalid: 4, 8, -2, 5.0, "prime", ...

    my_list: Annotated[list[int], 0:10]                 # Valid: [], [10, 20, 30, 40, 50]
                                                        # Invalid: (1, 2), ["abc"], [0] * 20
    your_set: Annotated[set[int], Len(0, 10)]           # Valid: {1, 2, 3}, ...
                                                        # Invalid: "Well, you get the idea!"

Documentation

While annotated-types avoids runtime checks for performance, users should not construct invalid combinations such as MultipleOf("non-numeric") or Annotated[int, Len(3)]. Downstream implementors may choose to raise an error, emit a warning, silently ignore a metadata item, etc., if the metadata objects described below are used with an incompatible type - or for any other reason!

Gt, Ge, Lt, Le

Express inclusive and/or exclusive bounds on orderable values - which may be numbers, dates, times, strings, sets, etc. Note that the boundary value need not be of the same type that was annotated, so long as they can be compared: Annotated[int, Gt(1.5)] is fine, for example, and implies that the value is an integer x such that x > 1.5. No interpretation is specified for special values such as nan.

We suggest that implementors may also interpret functools.partial(operator.le, 1.5) as being equivalent to Gt(1.5), for users who wish to avoid a runtime dependency on the annotated-types package.

To be explicit, these types have the following meanings:

  • Gt(x) - value must be "Greater Than" x - equivalent to exclusive minimum
  • Ge(x) - value must be "Greater than or Equal" to x - equivalent to inclusive minimum
  • Lt(x) - value must be "Less Than" x - equivalent to exclusive maximum
  • Le(x) - value must be "Less than or Equal" to x - equivalent to inclusive maximum

Interval

Interval(gt, ge, lt, le) allows you to specify an upper and lower bound with a single metadata object. None attributes should be ignored, and non-None attributes treated as per the single bounds above.

MultipleOf

MultipleOf(multiple_of=x) might be interpreted in two ways:

  1. Python semantics, implying value % multiple_of == 0, or
  2. JSONschema semantics, where int(value / multiple_of) == value / multiple_of.

We encourage users to be aware of these two common interpretations and their distinct behaviours, especially since very large or non-integer numbers make it easy to cause silent data corruption due to floating-point imprecision.

We encourage libraries to carefully document which interpretation they implement.

Len

Len() implies that min_inclusive <= len(value) < max_exclusive. We recommend that libraries interpret slice objects identically to Len(), making all the following cases equivalent:

  • Annotated[list, :10]
  • Annotated[list, 0:10]
  • Annotated[list, None:10]
  • Annotated[list, slice(0, 10)]
  • Annotated[list, Len(0, 10)]
  • Annotated[list, Len(max_exclusive=10)]

And of course you can describe lists of three or more elements (Len(min_inclusive=3)), four, five, or six elements (Len(4, 7) - note exclusive-maximum!) or exactly eight elements (Len(8, 9)).

Implementors: note that Len() should always have an integer value for min_inclusive, but slice objects can also have start=None.

Timezone

Timezone can be used with a datetime or a time to express which timezones are allowed. Annotated[datetime, Timezone[None]] must be a naive datetime. Timezone[...] (literal ellipsis) expresses that any timezone-aware datetime is allowed. You may also pass a specific timezone string or timezone object such as Timezone[timezone.utc] or Timezone["Africa/Abidjan"] to express that you only allow a specific timezone, though we note that this is often a symptom of fragile design.

Predicate

Predicate(func: Callable) expresses that func(value) is truthy for valid values. Users should prefer the statically inspectable metadata above, but if you need the full power and flexibility of arbitrary runtime predicates... here it is.

We provide a few predefined predicates for common string constraints: IsLower = Predicate(str.islower), IsUpper = Predicate(str.isupper), and IsDigit = Predicate(str.isdigit). Users are encouraged to use methods which can be given special handling, and avoid indirection like lambda s: s.lower().

Some libraries might have special logic to handle known or understandable predicates, for example by checking for str.isdigit and using its presence to both call custom logic to enforce digit-only strings, and customise some generated external schema.

We do not specify what behaviour should be expected for predicates that raise an exception. For example Annotated[int, Predicate(str.isdigit)] might silently skip invalid constraints, or statically raise an error; or it might try calling it and then propogate or discard the resulting TypeError: descriptor 'isdigit' for 'str' objects doesn't apply to a 'int' object exception. We encourage libraries to document the behaviour they choose.

Design & History

This package was designed at the PyCon 2022 sprints by the maintainers of Pydantic and Hypothesis, with the goal of making it as easy as possible for end-users to provide more informative annotations for use by runtime libraries.

It is deliberately minimal, and following PEP-593 allows considerable downstream discretion in what (if anything!) they choose to support. Nonetheless, we expect that staying simple and covering only the most common use-cases will give users and maintainers the best experience we can. If you'd like more constraints for your types - follow our lead, by defining them and documenting them downstream!

Comments
  • add GroupedMetadata as a base class for Interval

    add GroupedMetadata as a base class for Interval

    The main idea here is to generalize the pattern in Interval so that Pydantic and similar can define their Field as inheriting from GroupedMetadata, thus anything that can parse annotated-types will know how to unpack it (if it is not already unpacked by the PEP-646) even if it is a custom subclass in a library like Pydantic.

    Without this, some generic annotated-types parser would not know how to handle Pydantic's Field type without specific knowledge of Pydantic.

    opened by adriangb 19
  • I think we should remove `Regex`

    I think we should remove `Regex`

    I tried writing up Regex docs that would describe how people actually want to use them, and...

    Regex(regex_pattern=p, regex_flags=x) implies that the string should contain a match for p with flags x, at any position in the string. If you want the full string to match, or the match to be at the start or end of the string, you can use boundary markers like ^...$.

    Regex() can be used with unicode strings or byte strings; if either are allowed we suggest using one Regex() item for each type, and therefore ignoring Regex() items with the wrong string type.

    We do not specify the pattern syntax: libraries may choose to interpret it as the Python stdlib re module, regex package, ECMAscript syntax, etc., and we encourage them to clearly document their chosen interpretation. The meaning of the regex_flags argument is also implementation-defined.

    I think this is sufficiently-implementation-defined that we should just, well, make downstream implementations define their own Regex metadata with more precise semantics. Either that, or we explicitly separate PythonRegex(pattern: str|bytes, flags: int) from JSONschemaRegex(pattern: str) and make people choose one.

    opened by Zac-HD 14
  • Switch to hatchling & pip-tools

    Switch to hatchling & pip-tools

    poetry was annoying me, also it's lock file was out of date.

    Changes here:

    • use hatchling for build
    • use pip-compile (from pip-tools) to lock linting and testing requirements
    • tweak pre-commit
    • use pre-commit for linting
    • check github ref before release
    • add python 3.11 classifier
    • remove CI caching - seems to reduce CI time from ~1m20 to <20seconds, also simplifies CI setup
    opened by samuelcolvin 8
  • Magic syntax for constraints: `X > 5` -> `Gt(5)`, etc.

    Magic syntax for constraints: `X > 5` -> `Gt(5)`, etc.

    Closes #28; needs some more tests for validation errors and edge cases if we decide to go for it, and of course documentation.

    Importantly, adding most of the magic to our existing Interval and Len classes means that we can keep the API surface minimal, without exposing 'incomplete constraint' objects that don't really mean anything. Note that I've set this up to error out immediately if you try using the comparison magics on something which already represents a constraint. You can still do e.g. Interval(gt=3) < 5; this seems weird but OK to me in that it's very clear what it does and there's no duplication of bounds.

    opened by Zac-HD 6
  • ImportError: cannot import name 'Gt' from 'annotated_types'

    ImportError: cannot import name 'Gt' from 'annotated_types'

    Hi,

    I got an ImportError when I was trying to import Gt:

    ImportError: cannot import name 'Gt' from 'annotated_types'
        (/root/venv/lib/python3.9/site-packages/annotated_types/__init__.py)
    

    I tried installing and using the package on my personal computer (Python 3.9.12) and an EC2 instance (Python 3.9.13). The same error showed. I am wondering how to solve the problem.

    Thanks!

    opened by ychen878 6
  • use type aliases for shorthand constraints

    use type aliases for shorthand constraints

    https://github.com/annotated-types/annotated-types/pull/5#issuecomment-1118094109

    I think these names make more sense when they're wrapping the type, but I'm happy to keep the old ones

    opened by adriangb 6
  • Rethink `max_exclusive`, convert to `max_inclusive`?

    Rethink `max_exclusive`, convert to `max_inclusive`?

    If we have MaxLen, and (in pydantic at least) that can be set via a max_length argument.

    I really think this should mean "maximum inclusive", not "maximum exclusive" as currently documented.

    This matches (IMHO) much better people's assumption about what MaxLen(5) or max_length=5 or Len(0, 5) means:

    "The airbnb allows maximum 5 guests", you would assume 5 guests were allowed, not just 4

    If for the sake of correctness, that involves either:

    • treating slices differently
    • or, removing the recommendation on allowing slices

    That's sad, but I think a price worth paying.

    At the end of the way max_length=5 meaning any length up to 4, won't fly in pydantic.

    opened by samuelcolvin 5
  • Operator based `BaseMetadata` creation

    Operator based `BaseMetadata` creation

    Hi all, I really like the idea of BaseMetadata to formulate Annotated types.

    However, configuration using functions such as Gt(10) is not as easy as x > 10 to read. It would be nice to provide a non-constraint class All and support methods such as __gt__ like below.

    class All(BaseMetadata):
        def __gt__(self, other):
            return Gt(other)
    
    X = All()  # could be better to keep `All` private and provide `X` publically.
    Annotated[int, X > 5]  # equivalent to Annotated[int, Gt(5)]
    

    Similar logic can by applied to construct several objects.

    • Interval from 2 < X < 5
    • MultipleOf from X % 3 == 0
    • MaxLen from X.len() <= 10
    opened by hanjinliu 4
  • Chained constraints

    Chained constraints

    Thought from #28: we could also do something like Exponent = Annotated[int, Gt(0) & MultipleOf(2)].

    Implementing & would be easy: it just creates a GroupedMetadata.

    Other operators would be trickier: Annotated[str, Foo | Bar] should actually be expressed as Annotated[str, Bar] | Annotated[str, Foo], and I have no idea what XOR would be, I think we’d have to offload that to implementers. So I think this idea has short legs.

    opened by adriangb 2
  • Use `any()` instead of for loop & Invert `any/all` to simplify comparisons

    Use `any()` instead of for loop & Invert `any/all` to simplify comparisons

    Using Python's any() and all() built-in functions is a more concise way of doing this than using a for a loop.

    any() will return True when at least one of the elements evaluates to True, all() will return True only when all the elements evaluate to True.

    opened by yezz123 2
  • List of annotations vs. nested annotations

    List of annotations vs. nested annotations

    Hello, I was brought here from https://github.com/samuelcolvin/pydantic/discussions/4110.

    Perhaps this feature is already available or on the drawing-board, but I just wanted to say that it would be very useful if the user could provide a list of type-annotations like this:

    age: Annotated[int, [Gt(18), Lt(35)]]
    

    instead of having to recursively nest the annotations like this (although this should also be possible in case that might be useful in some applications):

    age: Annotated[Annotated[int, Gt(18)], Lt(35)]
    

    Thanks!

    opened by Hvass-Labs 2
Releases(v0.4.0)
  • v0.4.0(Oct 12, 2022)

    What's Changed

    • Switch to hatchling & pip-tools by @samuelcolvin in https://github.com/annotated-types/annotated-types/pull/22
    • convert Len to GroupedMetadata, add MinLen and MaxLen by @samuelcolvin in https://github.com/annotated-types/annotated-types/pull/21
    • switch from max_exclusive to max_length (inclusive) by @samuelcolvin in https://github.com/annotated-types/annotated-types/pull/24

    Full Changelog: https://github.com/annotated-types/annotated-types/compare/v0.3.1...v0.4.0

    Source code(tar.gz)
    Source code(zip)
  • v0.3.1(Sep 25, 2022)

    What's Changed

    • Add BaseMetadata to __all__ by @samuelcolvin in https://github.com/annotated-types/annotated-types/pull/19

    Full Changelog: https://github.com/annotated-types/annotated-types/compare/v0.3.0...v0.3.1

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Sep 25, 2022)

    What's Changed

    • Remove regex from tests by @adriangb in https://github.com/annotated-types/annotated-types/pull/13
    • add GroupedMetadata as a base class for Interval by @adriangb in https://github.com/annotated-types/annotated-types/pull/12
    • Remove regex from tests (again) by @adriangb in https://github.com/annotated-types/annotated-types/pull/14
    • use init_subclass instead of ABC by @adriangb in https://github.com/annotated-types/annotated-types/pull/16
    • add docs for GroupedMetadata and BaseMetadata by @adriangb in https://github.com/annotated-types/annotated-types/pull/15

    Full Changelog: https://github.com/annotated-types/annotated-types/compare/v0.2.0...v0.3.0

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jun 15, 2022)

PyTorch implementation of "Dataset Knowledge Transfer for Class-Incremental Learning Without Memory" (WACV2022)

Dataset Knowledge Transfer for Class-Incremental Learning Without Memory [Paper] [Slides] Summary Introduction Installation Reproducing results Citati

Habib Slim 5 Dec 05, 2022
Official Matlab Implementation for "Tiny Obstacle Discovery by Occlusion-aware Multilayer Regression", TIP 2020

Tiny Obstacle Discovery by Occlusion-aware Multilayer Regression Official Matlab Implementation for "Tiny Obstacle Discovery by Occlusion-aware Multil

Xuefeng 5 Jan 15, 2022
Drslmarkov - Distributionally Robust Structure Learning for Discrete Pairwise Markov Networks

Distributionally Robust Structure Learning for Discrete Pairwise Markov Networks

1 Nov 24, 2022
3D dataset of humans Manipulating Objects in-the-Wild (MOW)

MOW dataset [Website] This repository maintains our 3D dataset of humans Manipulating Objects in-the-Wild (MOW). The dataset contains 512 images in th

Zhe Cao 28 Nov 06, 2022
A simple Python configuration file operator.

A simple Python configuration file operator This project provides a common way to read configurations using config42. Installation It is possible to i

Scott Lau 2 Nov 08, 2021
Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding (CVPR2022)

Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding by Qiaole Dong*, Chenjie Cao*, Yanwei Fu Paper and Supple

Qiaole Dong 190 Dec 27, 2022
All public open-source implementations of convnets benchmarks

convnet-benchmarks Easy benchmarking of all public open-source implementations of convnets. A summary is provided in the section below. Machine: 6-cor

Soumith Chintala 2.7k Dec 30, 2022
Pytorch implementation of Value Iteration Networks (NIPS 2016 best paper)

VIN: Value Iteration Networks A quick thank you A few others have released amazing related work which helped inspire and improve my own implementation

Kent Sommer 297 Dec 26, 2022
Animation of solving the traveling salesman problem to optimality using mixed-integer programming and iteratively eliminating sub tours

tsp-streamlit Animation of solving the traveling salesman problem to optimality using mixed-integer programming and iteratively eliminating sub tours.

4 Nov 05, 2022
Code To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment.

COLIEE 2021 - task 2: Legal Case Entailment This repository contains the code to reproduce NeuralMind's submissions to COLIEE 2021 presented in the pa

NeuralMind 13 Dec 16, 2022
ProMP: Proximal Meta-Policy Search

ProMP: Proximal Meta-Policy Search Implementations corresponding to ProMP (Rothfuss et al., 2018). Overall this repository consists of two branches: m

Jonas Rothfuss 212 Dec 20, 2022
ETMO: Evolutionary Transfer Multiobjective Optimization

ETMO: Evolutionary Transfer Multiobjective Optimization To promote the research on ETMO, benchmark problems are of great importance to ETMO algorithm

Songbai Liu 0 Mar 16, 2021
Gans-in-action - Companion repository to GANs in Action: Deep learning with Generative Adversarial Networks

GANs in Action by Jakub Langr and Vladimir Bok List of available code: Chapter 2: Colab, Notebook Chapter 3: Notebook Chapter 4: Notebook Chapter 6: C

GANs in Action 914 Dec 21, 2022
Code for "LoFTR: Detector-Free Local Feature Matching with Transformers", CVPR 2021

LoFTR: Detector-Free Local Feature Matching with Transformers Project Page | Paper LoFTR: Detector-Free Local Feature Matching with Transformers Jiami

ZJU3DV 1.4k Jan 04, 2023
Recursive Bayesian Networks

Recursive Bayesian Networks This repository contains the code to reproduce the results from the NeurIPS 2021 paper Lieck R, Rohrmeier M (2021) Recursi

Robert Lieck 11 Oct 18, 2022
This repository contains code from the paper "TTS-GAN: A Transformer-based Time-Series Generative Adversarial Network"

TTS-GAN: A Transformer-based Time-Series Generative Adversarial Network This repository contains code from the paper "TTS-GAN: A Transformer-based Tim

Intelligent Multimodal Computing and Sensing Laboratory (IMICS Lab) - Texas State University 108 Dec 29, 2022
Volumetric parameterization of the placenta to a flattened template

placenta-flattening A MATLAB algorithm for volumetric mesh parameterization. Developed for mapping a placenta segmentation derived from an MRI image t

Mazdak Abulnaga 12 Mar 14, 2022
Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments Paper: arXiv (ICRA 2021) Video : https://youtu.be/CC

Sachini Herath 68 Jan 03, 2023
The PyTorch implementation of paper REST: Debiased Social Recommendation via Reconstructing Exposure Strategies

REST The PyTorch implementation of paper REST: Debiased Social Recommendation via Reconstructing Exposure Strategies. Usage Download dataset Download

DMIRLAB 2 Mar 13, 2022
official code for dynamic convolution decomposition

Revisiting Dynamic Convolution via Matrix Decomposition (ICLR 2021) A pytorch implementation of DCD. If you use this code in your research please cons

Yunsheng Li 110 Nov 23, 2022