Pre-commit hook for upgrading type hints

Overview

supported python versions code coverage pre-commit

PEP585 Upgrade

This is a pre-commit hook configured to automatically upgrade your type hints to the new native types implemented in PEP 585.

This will work for any Python version above 3.7, though if you're not using 3.9 you will need to run the hook with futures-imports=true.

A complete type list is shown below.

See the complete list
Used to be Will be upgraded to
typing.Tuple tuple
typing.List list
typing.Dict dict
typing.Set set
typing.FrozenSet frozenset
typing.Type type
typing.Deque collections.deque
typing.DefaultDict collections.defaultdict
typing.OrderedDict collections.OrderedDict
typing.Counter collections.Counter
typing.ChainMap collections.ChainMap
typing.Awaitable collections.abc.Awaitable
typing.Coroutine collections.abc.Coroutine
typing.AsyncIterable collections.abc.AsyncIterable
typing.AsyncIterator collections.abc.AsyncIterator
typing.AsyncGenerator collections.abc.AsyncGenerator
typing.Iterable collections.abc.Iterable
typing.Iterator collections.abc.Iterator
typing.Generator collections.abc.Generator
typing.Reversible collections.abc.Reversible
typing.Container collections.abc.Container
typing.Collection collections.abc.Collection
typing.Callable collections.abc.Callable
typing.AbstractSet collections.abc.Set
typing.MutableSet collections.abc.MutableSet
typing.Mapping collections.abc.Mapping
typing.MutableMapping collections.abc.MutableMapping
typing.Sequence collections.abc.Sequence
typing.MutableSequence collections.abc.MutableSequence
typing.ByteString collections.abc.ByteString
typing.MappingView collections.abc.MappingView
typing.KeysView collections.abc.KeysView
typing.ItemsView collections.abc.ItemsView
typing.ValuesView collections.abc.ValuesView
typing.ContextManager contextlib.AbstractContextManager
typing.AsyncContextManager contextlib.AbstractAsyncContextManager
typing.re.Pattern re.Pattern
typing.re.Match re.Match

I'm a visual learner

In a nutshell, this code

from typing import List, Tuple, Dict, Set, FrozenSet

def do_thing(x: List[Tuple[str, ...]], y: Dict[str, Set[str]]) -> FrozenSet:

becomes this

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

or this, if you're running python < 3.9 or enable the futures option

from __future__ import annotations

def do_thing(x: list[tuple[str, ...]], y: dict[str, set[str]]) -> frozenset:

Features:

  • Performs in-line substitution for new types
  • Adds new imports for upgrade types which need them
  • Adds __futures__ imports if the futures flag is enabled
  • Removes no longer needed typing imports

Note: even though we remove and add imports reasonably well, I would recommend running this in tandem with hooks like isort to aggregate and sort your imports, and flake8 to discover any unused imports neither were able to remove. Otherwise you risk needing to do some manual cleanup from time to time (though it should be pretty rare).

Config

To use this with pre-commit, simply add this to your config file:

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints

and while futures imports are added automatically if you're running Python older than 3.9, you can also enable them explicitly, by adding a --futures arg.

This is required, e.g., when maintaining code that needs to support older Python versions.

- repo: https://github.com/sondrelg/pep585-upgrade
  rev: ''  # Use the sha / tag you want to point at
  hooks:
  - id: upgrade-type-hints
    args: [ '--futures=true' ]

For more information about available arguments, see the function definitions.

Running this once on my codebase

If you wish to run this once on your codebase, that's not easy to do without pre-commit, as it piggybacks on that process quite a bit.

However, installing pre-commit and configuring the hook to run will take you less than a minute. These are the steps:

  • Run pip install pre-commit
  • Run touch .pre-commit-config.yaml
  • Copy the configuration shown above into the file
  • Run pre-commit run --all-files

Running this once on a single file

To run the upgrade on a single file, you can clone the repo and run python -m upgrade_type_hints from the src folder, or something equivalent.

Known imperfections

  • We have a hard time removing common import typing imports, since we don't have a full inventory of all possible places typing could be used. Seeing something like this, you might think this is easy to handle

    import typing
    
    x: typing.List

    but extending this example to a thousand-line file, the way we've structured the logic, there is no way to know whether there is a valid typing.Optional somewhere in the file.

  • We might remove typing imports in a file where you needed them for more than just type annotations. An example of this is custom type declarations:

    from typing import List
    
    x: List  # this will be upgraded and the import will be removed
    y = List[str]  # this will be left without its required import

    The reason for this is that custom type declarations are not a part of the ast objects we look at.

Both points are resolved by running flake8.

Supporting the project

Please leave a ✭ if this project helped you 👏 and contributions are always welcome!

Comments
  • Don't upgrade generics - part 2

    Don't upgrade generics - part 2

    I observe the same issue as #22, using v1.0.1.

    Starting with the file

    from __future__ import annotations
    
    from typing import Any, Callable, Mapping
    
    MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    def func(x: Mapping[str, Any], y: Any) -> None:
        return None
    
    callback: MessageCallback
    callback = func
    

    This originally runs without errors, but not after running pep585-upgrade has run:

    $ python -V
    Python 3.8.6
    $ python test.py
    $ pre-commit run
    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    - files were modified by this hook
    
    Fixing src/workflows/test.py
    
    isort....................................................................Failed
    - hook id: isort
    - files were modified by this hook
    
    $ git diff
    index 72539fc..56bc282 100644
    --- a/test.py
    +++ b/test.py
    @@ -1,6 +1,7 @@
     from __future__ import annotations
    
    -from typing import Any, Callable, Mapping
    +from collections.abc import Mapping
    +from typing import Any, Callable
    
     MessageCallback = Callable[[Mapping[str, Any], Any], None]
    
    $ python test.py
    Traceback (most recent call last):
      File "test.py", line 6, in <module>
        MessageCallback = Callable[[Mapping[str, Any], Any], None]
    TypeError: 'ABCMeta' object is not subscriptable
    
    opened by Anthchirp 11
  • Support for typing.cast call

    Support for typing.cast call

    The code-base I'm modernizing has the following code:

    project_acl_list = cast(Optional[List[str]], project.acl)
    

    Unfortunately, this is not transformed.

    I'm willing to fix that, but I don't know yet how...

    Could we discuss how to implement the transformation for this code?

    opened by rominf 9
  • end_lineno error while running python 3.7

    end_lineno error while running python 3.7

    Adding hook via:

      - repo: https://github.com/sondrelg/pep585-upgrade
        rev: ''
        hooks:
          - id: upgrade-type-hints
            args: [ '--futures=true' ]
    

    When running in a python 3.9 virtualenv everything works, but switching to python 3.7 causes this error:

    Upgrade type hints.......................................................Failed
    - hook id: upgrade-type-hints
    - exit code: 1
    
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    Traceback (most recent call last):
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/bin/upgrade-type-hints-script", line 8, in <module>
        sys.exit(main())
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/main.py", line 33, in main
        annotation_list, imports, futures_import_found = find_annotations_and_imports_in_file(filename)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 128, in find_annotations_and_imports_in_file
        imports, futures_import_found = map_imports(tree)
      File "/home/some_name/.cache/pre-commit/repoh9mucjmj/py_env-python3.7/lib/python3.7/site-packages/src/upgrade_type_hints/checker.py", line 113, in map_imports
        imports[item.lineno]['end_lineno'] = item.end_lineno
    AttributeError: 'Import' object has no attribute 'end_lineno'
    
    opened by ArcLightSlavik 8
  • pre-commit autoupdate warning

    pre-commit autoupdate warning

    Running pre-commit autoupdate leads to the following result: Updating https://github.com/sondrelg/pep585-upgrade ... updating v1.0.1 -> v1.

    Running pre-commit autoupdate again will result in the following warning:

    [WARNING] The 'rev' field of repo 'https://github.com/sondrelg/pep585-upgrade' appears to be a mutable reference (moving tag / branch).
    Mutable references are never updated after first install and are not supported. 
    See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details. 
    Hint: `pre-commit autoupdate` often fixes this.
    

    It would be cool if pre-commit autoupdate would work on this hook as expected. I suspect that the problem is that all 3 tags point to the same commit.

    opened by alkatar21 7
  • Consider module docstring while inserting future import

    Consider module docstring while inserting future import

    Summary

    When the python file consists of module docstring, the formatter inserts the from __future__ import annotation on the top of the docstring.

    Previous Behavior

    Source

    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    from typing import List
    
    a: List[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
        pass
    

    Generated

    from __future__ import annotations
    
    
    #! /usr/env/bin python
    """don't touch the doc string
    
    multi line boss
    """
    
    a: list[int] = [1, 2]
    
    
    def foo():
        """I'm a doc string"""
    

    The PR adds a function to find the module docstring. It will insert future import after the module docstring.

    opened by kracekumar 4
  • Shebang is not respected

    Shebang is not respected

    Input:

    #!/usr/bin/env python3
    from typing import AbstractSet
    
    s: AbstractSet
    

    Output:

    from collections.abc import Set
    #!/usr/bin/env python3
    
    s: Set
    

    I didn't prepare the fix yet, but I think that the best solution would be to save the first occurrence of non-comment, non-docstring expressions while scanning AST and put the imports before that. What do you think?

    opened by rominf 4
  • Create/push a tag for the package

    Create/push a tag for the package

    Hi guys, Thanks a lot for this great package, it makes the code much more pythonic! I have one favor: could you create/push a tag for the package? Else I have this annoying warning:

    [WARNING] The 'rev' field of repo 'https://github.com/snok/pep585-upgrade' appears to be a mutable reference (moving tag / branch).  Mutable references are never updated after first install and are not supported.  See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.
    
    opened by MRigal 3
  • don't add another import line from same package

    don't add another import line from same package

    For an input file like this:

    from collections.abc import Callable
    
    from typing import Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    when applying pep585 hook I get:

    from collections.abc import Collection
    from collections.abc import Callable
    
    def a(a: Callable, b: Collection):
        pass
    

    while I expect:

    from collections.abc import Callable, Collection
    
    def a(a: Callable, b: Collection):
        pass
    

    It could be argued this is the job of e.g. isort, however - pep585-upgrade already tries to be nice in terms of imports - e.g. in a larger import set - this new import line gets added nearby other import from same package and not at the end (which would be simplest I guess)

    opened by kretes 1
  • Don't Upgrade Generics

    Don't Upgrade Generics

    The typing package supports generics, whereas collections.abc does not. Hence, when this tool replaces typing types that are used as generics with their equivalent types in collections.abc, it breaks the build. I found it helpful to run this tool as a one-off and then manually undo these cases though, so thank you!

    opened by Kurt-von-Laven 1
  • Imports removal corrupts imports

    Imports removal corrupts imports

    Minimal working example:

    Input:

    from typing import Any, Type, TypeVar
    
    t: Type
    

    Output:

    from typing import AnyVar
    
    t: type
    

    I have a fix for this already. Please wait for a PR with a fix and tests.

    opened by rominf 1
  • Correct name of .pre-commit-config.yaml

    Correct name of .pre-commit-config.yaml

    I got an error from pre-commit which suggested it was looking for a .pre-commit-config.yaml file, when I changed the filename, as in this commit.

    Possibly OS specific, this was on MacOS Big Sur, Python 3.9.

    opened by james-d-f 1
Releases(v1.0.1)
Owner
snok
Open source collaboration organization made by @jonasks & @sondrelg
snok
Camera track the tip of a pen to use as a drawing tablet

cablet Camera track the tip of a pen to use as a drawing tablet Setup You will need: Writing utensil with a colored tip (preferably blue or green) Bac

14 Feb 20, 2022
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

project: Faros 9 Jul 18, 2022
Minterpy - Multidimensional interpolation in Python.

minterpy is an open-source Python package for a multivariate generalization of the classical Newton and Lagrange interpolation schemes as well as related tasks.

Center for Advanced Systems Understanding 18 Jan 06, 2023
Sorter makes file organisation and management easier.

Sorter Sorter makes file organisation easier. It simply helps you organise several files that contain similar characteristics into a single folder. Yo

Aswa Paul 34 Aug 14, 2022
Algo próximo do ARP

ArpPY Algo parecido com o ARP-Scan. Dependencias O script necessita no mínimo ter o Python versão 3.x instalado e ter o sockets instalado. Executando

Feh's 3 Jan 18, 2022
Quantity Takeoff with Python. Collecting groups of elements by filters

The free tool QuantityTakeoff allows you to group elements from Revit and IFC models (in BIMJSON-CSV format) with just a few filters and find the required volume values for the grouped elements.

OpenDataBIM 9 Jan 06, 2023
An almost fully customizable language made in python!

Whython is a project language, the idea of it is that anyone can download and edit the language to make it suitable to what they want.

Julian 47 Nov 05, 2022
The git for the Python Story Utility Package library.

PSUP, The Python Story Utility Package Module. PSUP helps making stories or games with options, diverging paths, different endings and so on. You can

Enoki 6 Nov 27, 2022
Simple dotfile pre-processor with a per-file configuration

ix (eeks) Simple dotfile pre-processor with a per-file configuration Summary (TL;DR) ix.py is all you need config is an ini file. files to be processe

Poly 12 Dec 16, 2021
Participants of Bertelsmann Technology Scholarship created an awesome list of resources and they want to share it with the world, if you find illegal resources please report to us and we will remove.

Participants of Bertelsmann Technology Scholarship created an awesome list of resources and they want to share it with the world, if you find illegal

Wissem Marzouki 29 Nov 28, 2022
WGGCommute - Adding Commute Times to WG-Gesucht Listings

WGGCommute - Adding Commute Times to WG-Gesucht Listings This is a barebones implementation of a chrome extension that can be used to add commute time

Jannis 2 Jul 20, 2022
A New, Interactive Approach to Learning Python

This is the repository for The Python Workshop, published by Packt. It contains all the supporting project files necessary to work through the course from start to finish.

Packt Workshops 231 Dec 26, 2022
Advanced IPv4 Subnet Calculator in Python3

Advanced IPv4 Subnet Calculator in Python3 Table of Contents Getting Started Installation How it works? SVI Configuration Template Previews Getting St

Osama Abbas 1 May 10, 2022
Attempt at a Windows version of the plotman Chia Plot Manager system

windows plotman: an attempt to get plotman to work on windows THIS IS A BETA. Not ready for production use just yet. Almost, but not quite there yet.

59 May 11, 2022
Py-Parser est un parser de code python en python encore en plien dévlopement.

PY - PARSER Py-Parser est un parser de code python en python encore en plien dévlopement. Une fois achevé, il servira a de nombreux projets comme glad

pf4 3 Feb 21, 2022
Automatización del proceso Inmofianza

Selenium Inmofianza Proyecto de pruebas automatizadas con selenium webdriver para el aplicativo Omnicanalidad Pre-requisitos 📋 Componentes que deben

Natalia Narváez 1 Jan 07, 2022
DNA Storage Simulator that analyzes and simulates DNA storage

DNA Storage Simulator This monorepository contains code for a research project by Mayank Keoliya and supervised by Djordje Jevdjic, that analyzes and

Mayank Keoliya 3 Sep 25, 2022
1 May 12, 2022
A simple wrapper to analyse and visualise reinforcement learning agents' behaviour in the environment.

Visrl Visrl (pronounced "visceral") is a simple wrapper to analyse and visualise reinforcement learning agents' behaviour in the environment. Reinforc

Jet New 14 Jun 27, 2022