This is a survey of python's async concurrency features by example.

Overview

Survey of Python's Async Features

This is a survey of python's async concurrency features by example.

The purpose of this survey is to demonstrate that using Python's async features isn't difficult, and they are simple to use in all IO bound contexts. They can be used simply and to great effect in scripts, existing sync backends, ETLs, wherever.

Most examples necessarily demonstrate how to interop with synchronous code, because all python programs begin in a blocking main thread. The event loop must be explicitly evoked and given control of the main thread. The interop examples become more useful in later parts.

Sync/Async interop is particularly necessary in a language that introduced async features after the fact. Python supplies several ways for async code to easily await on blocking functions (from, for instance, db drivers that block on io) without halting the event loop. Some of these techniques are covered in later sections.

When to use these features?

Whenever the program's performance is constrained by the performance of the hosts I/O subsystem. In other words, I/O bound contexts. This is usually contrasted with CPU bound applications, where the applications performance is dependent on the speed of the CPU. If the CPU in question has multple cores, the event loop, having a single execution thread, cannot execute multiple coroutines in parallel.

However, Python's async features could still be useful in dispatching and coordinating work between coroutines and different process workers, which can take advantage of multi core CPU's. Python provides some useful abstractions (e.g. Executors) which are designed to interop between the event loop and python threads/processes.

Why not use python multithreading for an IO Bound application?

NOTE: Generally, the GIL ensures that only one thread can execute at once.

  1. There is effectively an event loop anyway, but the operating system decides which thread should execute and for how long.
  2. The operating system doesn't understand how the threads should cooperate. Therefore opens the possiblity of race conditions and unsafe operations on memory unless the program accounts for the unpredictable nature of threaded execution (via locks, queues, etc.). In otherwords, the program has to be written to cooperate after the fact, versus an environment which is inherently cooperative.
  3. Threads have overhead.

Versus async concurrency:

  1. The programmer controls context switching between execution units.
  2. The event loop executes in a single thread, and the coroutines must explicitly yield back control making race conditions impossible. Also, since only one coroutine can execute at once, memory access and mutation is incidentally atomic.
  3. One call stack, versus one per thread (in additon to other overhead from OS thread). Also, exceptions are easier to trace, because there is one stack.

Setup

make build
make server
# new tab or terminal window
make part_<01-16>

Index

Event Loop

  1. Obtaining or creating the event loop

  2. Scheduling and running a coroutine to completion on the event loop

Awaitables

  1. Coroutines

  2. Futures

  3. Tasks

  4. Awaitable Objects

  5. Async Context Managers

  6. Async Iterators

  7. Async Generators

Concurrency

  1. Via loop.create_task

  2. Via asyncio.gather (asyncio.sleep example)

  3. Via asyncio.gather (aiohttp example)

  4. Via asyncio.as_completed

Sync / Async Interop

  1. loop.run_in_executor (thread pool, io bound)

  2. loop.run_in_executor (process pool, cpu bound)

Extras

  1. async queue consumer / producer

  2. asyncio Stream API - mini redis

Owner
Tyler Lovely
Tyler Lovely
Automated Changelog/release note generation

Quickly generate changelogs and release notes by analysing your git history. A tool written in python, but works on any language.

Documatic 95 Jan 03, 2023
Load dependent libraries dynamically.

dypend dypend Load dependent libraries dynamically. A few days ago, I encountered many users feedback in an open source project. The Problem is they c

Louis 5 Mar 02, 2022
Rick Astley Language is a rick roll oriented, dynamic, strong, esoteric programming language.

Rick Roll Language / Rick Astley Language A rick roll oriented, dynamic, strong, esoteric programming language. Prolegomenon The reasons that I made t

Rick Roll Programming Language 658 Jan 09, 2023
Painel simples com consulta de cep,CNPJ,placa e ip

Painel mpm Um painel simples com consultas de IP, CNPJ, CEP, PLACA, TELEFONE, CPF e NOME Início 🌐 apt update && apt upgrade -y pkg i python git pip i

8 Feb 27, 2022
Practice in Oxford_AI&ML class

Practice in Oxford_AI&ML class

St3ve Lee 2 Feb 04, 2022
Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as

Barthelemy Dagenais 1k Jan 02, 2023
This is a database of 180.000+ symbols containing Equities, ETFs, Funds, Indices, Futures, Options, Currencies, Cryptocurrencies and Money Markets.

Finance Database As a private investor, the sheer amount of information that can be found on the internet is rather daunting.

Jeroen Bouma 1.4k Dec 31, 2022
Python 3.9.4 Graphics and Compute Shader Framework and Primitives with no external module dependencies

pyshader Python 3.9.4 Graphics and Compute Shader Framework and Primitives with no external module dependencies Fully programmable shader model (even

Alastair Cota 1 Jan 11, 2022
GUI for the Gammu library.

Wammu GUI for the Gammu library. Homepage https://wammu.eu/ License GNU GPL version 3 or later. First start On first start you will be asked for set

Gammu 60 Dec 14, 2022
Prints values and types during compilation!

Compile-Time Printer Compile-Time Printer prints values and types at compile-time in C++. Teaser test.cpp compile-time-printer

43 Dec 26, 2022
A powerful and user-friendly binary analysis platform!

angr angr is a platform-agnostic binary analysis framework. It is brought to you by the Computer Security Lab at UC Santa Barbara, SEFCOM at Arizona S

6.3k Jan 02, 2023
A pet facts python api

Pet-Facts-API A pet facts python api Project Links API :- https://pet-facts-api.vercel.app Docs :- https://fayasnoushad.github.io/Pet-Facts-API

Fayas Noushad 3 Dec 18, 2021
Socorro is the Mozilla crash ingestion pipeline. It accepts and processes Breakpad-style crash reports. It provides analysis tools.

Socorro Socorro is a Mozilla-centric ingestion pipeline and analysis tools for crash reports using the Breakpad libraries. Support This is a Mozilla-s

Mozilla Services 552 Dec 19, 2022
Scripts used in the RayStation medical radiation dosimetry treatment planning system

Med Phys Scripts These are scripts that I, the medical physics assistant at Cookeville Regional Medical Center, wrote for use in our radiation therapy

Kaley White 2 Oct 19, 2022
Stop ask your soraka to ult you, just ult yourself

Lollo's auto-ultimate script Are you tired of your low elo friend who can't ult you with soraka when you ask for it? Use Useless Support and just ult

9 Oct 20, 2022
Automatización del proceso Inmofianza

Selenium Inmofianza Proyecto de pruebas automatizadas con selenium webdriver para el aplicativo Omnicanalidad Pre-requisitos 📋 Componentes que deben

Natalia Narváez 1 Jan 07, 2022
PatZilla is a modular patent information research platform and data integration toolkit with a modern user interface and access to multiple data sources.

PatZilla is a modular patent information research platform and data integration toolkit with a modern user interface and access to multiple data sources.

IP Tools 68 Dec 14, 2022
Solves Maths24 problems for you!

maths24-solver Solves Maths24 problems for you! Enjoy this open scource project! You can edit modify and share! My wishes is for you to use this proje

6 Nov 07, 2021
Simple dotfile pre-processor with a per-file configuration

ix (eeks) Simple dotfile pre-processor with a per-file configuration Summary (TL;DR) ix.py is all you need config is an ini file. files to be processe

Poly 12 Dec 16, 2021
Incident Response Process and Playbooks | Goal: Playbooks to be Mapped to MITRE Attack Techniques

PURPOSE OF PROJECT That this project will be created by the SOC/Incident Response Community Develop a Catalog of Incident Response Playbook for every

Austin Songer 987 Jan 02, 2023