This is a survey of python's async concurrency features by example.

Overview

Survey of Python's Async Features

This is a survey of python's async concurrency features by example.

The purpose of this survey is to demonstrate that using Python's async features isn't difficult, and they are simple to use in all IO bound contexts. They can be used simply and to great effect in scripts, existing sync backends, ETLs, wherever.

Most examples necessarily demonstrate how to interop with synchronous code, because all python programs begin in a blocking main thread. The event loop must be explicitly evoked and given control of the main thread. The interop examples become more useful in later parts.

Sync/Async interop is particularly necessary in a language that introduced async features after the fact. Python supplies several ways for async code to easily await on blocking functions (from, for instance, db drivers that block on io) without halting the event loop. Some of these techniques are covered in later sections.

When to use these features?

Whenever the program's performance is constrained by the performance of the hosts I/O subsystem. In other words, I/O bound contexts. This is usually contrasted with CPU bound applications, where the applications performance is dependent on the speed of the CPU. If the CPU in question has multple cores, the event loop, having a single execution thread, cannot execute multiple coroutines in parallel.

However, Python's async features could still be useful in dispatching and coordinating work between coroutines and different process workers, which can take advantage of multi core CPU's. Python provides some useful abstractions (e.g. Executors) which are designed to interop between the event loop and python threads/processes.

Why not use python multithreading for an IO Bound application?

NOTE: Generally, the GIL ensures that only one thread can execute at once.

  1. There is effectively an event loop anyway, but the operating system decides which thread should execute and for how long.
  2. The operating system doesn't understand how the threads should cooperate. Therefore opens the possiblity of race conditions and unsafe operations on memory unless the program accounts for the unpredictable nature of threaded execution (via locks, queues, etc.). In otherwords, the program has to be written to cooperate after the fact, versus an environment which is inherently cooperative.
  3. Threads have overhead.

Versus async concurrency:

  1. The programmer controls context switching between execution units.
  2. The event loop executes in a single thread, and the coroutines must explicitly yield back control making race conditions impossible. Also, since only one coroutine can execute at once, memory access and mutation is incidentally atomic.
  3. One call stack, versus one per thread (in additon to other overhead from OS thread). Also, exceptions are easier to trace, because there is one stack.

Setup

make build
make server
# new tab or terminal window
make part_<01-16>

Index

Event Loop

  1. Obtaining or creating the event loop

  2. Scheduling and running a coroutine to completion on the event loop

Awaitables

  1. Coroutines

  2. Futures

  3. Tasks

  4. Awaitable Objects

  5. Async Context Managers

  6. Async Iterators

  7. Async Generators

Concurrency

  1. Via loop.create_task

  2. Via asyncio.gather (asyncio.sleep example)

  3. Via asyncio.gather (aiohttp example)

  4. Via asyncio.as_completed

Sync / Async Interop

  1. loop.run_in_executor (thread pool, io bound)

  2. loop.run_in_executor (process pool, cpu bound)

Extras

  1. async queue consumer / producer

  2. asyncio Stream API - mini redis

Owner
Tyler Lovely
Tyler Lovely
JSEngine is a simple wrapper of Javascript engines.

JSEngine This is a simple wrapper of Javascript engines, it wraps the Javascript interpreter for Python use. There are two ways to call interpreters,

11 Dec 18, 2022
A program that lets you use your tablet's tilting to emulate an actual joystick on a Linux computer.

Tablet Tilt Joystick A program that lets you use your tablet's tilting to emulate an actual joystick on a Linux computer. It's called tablet tilt joys

1 Feb 07, 2022
Scientific color maps and standardization tools

Scicomap is a package that provides scientific color maps and tools to standardize your favourite color maps if you don't like the built-in ones. Scicomap currently provides sequential, bi-sequential

Thomas Bury 14 Nov 30, 2022
A country information finder module

A country information finder module

Fayas Noushad 3 Nov 28, 2021
TallerStereoVision Convencion Python Chile 2021

TallerStereoVision Convencion Python Chile 2021 Taller Stereo Vision & Python PyCon.cl 2021 InstalaciĂłn Se recomienta utilizar Virtual Environment pyt

2 Oct 20, 2022
How to access and display MyEnergi data

MyEnergi-Python-Example How to access and display MyEnergi data Windows PC Install a version of Python typically 3.10 The Python code here needs addit

G6EJD 8 Nov 28, 2022
Pomodoro timer by the Algodrip team!

PomoDrip 🍅 Pomodoro timer by the Algo Drip team! To-do: Create the script for the pomodoro timer Design the front-end of the program (Flask or Javasc

Algodrip 3 Sep 12, 2021
Singularity Containers on Apple M1 (ARM64)

Singularity Containers on Apple M1 (ARM64) This is a repository containing a ready-to-use environment for singularity in arm64 (M1). It has been prepa

Manuel Parra 4 Nov 14, 2022
Feature engineering library that helps you keep track of feature dependencies, documentation and schema

Feature engineering library that helps you keep track of feature dependencies, documentation and schema

28 May 31, 2022
A Way to Use Python, Easier.

PyTools A Way to Use Python, Easier. How to Install Just copy this code, then make a new file in your project directory called PyTools.py, then paste

Kamran 2 Aug 15, 2022
Tucan Discord Token Generator - Remastered

TucanGEN-SRC Tucan Discord Token Generator - Remastered Tucan source made better by me. -- idk if it works anymore Includes: hCaptcha Bypass Automatic

Vast 8 Nov 04, 2022
Python library for converting Python calculations into rendered latex.

Covert art by Joshua Hoiberg handcalcs: Python calculations in Jupyter, as though you wrote them by hand. handcalcs is a library to render Python calc

Connor Ferster 5.1k Jan 07, 2023
About Python's multithreading and GIL

About Python's multithreading and GIL

Souvik Ghosh 3 Mar 01, 2022
Heads Down Application for Mac OSX

Heads Down A Mac app that lives in your ribbon—with a click of the mouse, temporarily block distracting websites and applications to encourage "heads

20 Mar 10, 2021
A git extension for seeing your Cloud Build deployment

A git extension for seeing your Cloud Build deployment

Katie McLaughlin 13 May 10, 2022
This Program Automates The Procces Of Adding Camos On Guns And Saving Them On Modern Warfare Guns

This Program Automates The Procces Of Adding Camos On Guns And Saving Them On Modern Warfare Guns

Flex Tools 6 May 26, 2022
Camera track the tip of a pen to use as a drawing tablet

cablet Camera track the tip of a pen to use as a drawing tablet Setup You will need: Writing utensil with a colored tip (preferably blue or green) Bac

14 Feb 20, 2022
HogwartsRegister - A Hogwarts Register With Python

A Hogwarts Register Installation download code git clone https://github.com/haor

0 Feb 12, 2022
Laurence Billingham 1 Feb 16, 2022
A minimalist starknet amm adapted from StarkWare's amm.

viscus • A minimalist starknet amm adapted from StarkWare's amm. Directory Structure contracts

Alucard 4 Dec 27, 2021