CPython extension implementing Shared Transactional Memory with native-looking interface

Overview

CPython extension implementing Shared Transactional Memory with native-looking interface

This is my alternative to channels and pickle for cross-interpreter communication for the ongoing PEP 554 -- Multiple Interpreters in the Stdlib and multi-core-python. Treat this project as a deep rework of a standard multiprocessing.sharedctypes, intended to implement a support for complex dynamic data and complex atomic operations.

See INSTALL.txt for instructions on setup. Only 32-bit Linux or Windows are supported, and probably CPython 3.7+. It's still kinda proof-of-concept, so don't expect too much from it.

https://habr.com/en/post/585320/ - Detailed article with examples, benchmarks, and review of implementation details.

Brief tutorial on usage

  1. Import the library using import pso;
  2. Primary process calls pso.init() and notes the returned unique ID of the session;
  3. Secondary processes join the session with pso.connect(ID);
  4. Now all the processes can access a common pso.root() storage via attributes of this object.

Currently the pso.root() shared storage can be used for such types as: str, bytes, bool, int, float, dict, list, tuple, instances of pso.ShmObject and its subclasses, and any combination of mentioned data types.

For example, you can run this code side by side in two terminals:

Primary Secondary
>>> import pso >>> import pso
>>> pso.init()
'pso_FHu0a4idD'
>>> pso.connect('pso_FHu0a4idD')
>>> r = pso.root()
>>> r.list = []
r = pso.root()
>>> r.list
ShmList: 0 elements , empty
>>> r.list.append('stringy')
>>> r.list
ShmList: 1 elements , first element: stringy
>>> r.list
ShmList: 1 elements , first element: stringy

Those are basics you can do with multiprocessing.ShareableList too... except you cannot grow the list after creation and you cannot put other mutable containers into it. But we are just getting started.

Transactions

Extending the two terminals example:

Primary Secondary
>>> pso.transaction_start()
>>> r.list.append('another string')
>>> len(r.list)
1
>>> pso.transaction_commit()
>>> len(r.list)
2
>>> pso.transaction_start()
>>> r.list.append('third string')
>>> r.list
execution pauses
>>> pso.transaction_commit() ShmList: 3 elements , first element: stringy

As you can see, the changes made within transaction are invisible to other processes until committed. Moreover, modified uncommitted objects cannot be read by other processes. By making multiple changes withing a single transaction, you can commit them atomically and be sure nobody sees half-committed inconsistent data.

At first it might look like a simple fine-grained locking, but actually it magically resolves deadlocks and resource starvation. Let's repeat the last operation again but do the reading part with a longer running transaction:

Primary Secondary
>>> pso.transaction_start()
>>> r.list
ShmList: 3 elements , first element: stringy
>>> pso.transaction_start()
>>> r.list.append('fourth string')
Traceback (most recent call last):
File "", line 1, in
pso.ShmAbort: Transaction aborted

Oops, what happenned? The two transactions contended for the same resource again, but the result is different. Obviously, one transaction should be terminated and the other should keep running. PSO knows which one of them is running longer and allows it to proceed, while a recently started transaction is aborted.

In this very example the primary process is left in a limbo state where you are required to handle the contention somehow e.g. do pso.transaction_abort(). However, there already exist functions that can handle everything for you. Those are: pso.transaction() which accepts a function to be executed atomically, and a special with transaction: syntactic sugar which can run an unnamed block atomically e.g.:

with transaction:
    r.list.append('Fifth element')

This syntactic sugar requires a special module loader, activated either by running python3 -m pso modulename.py or by naming a loaded module with a .pso suffix, like modulename.pso.py. The latter option is only available once the "pso" module is loaded, so it won't work for the main module of your project.

You can find example programs in the examples/ folder, those are launched from the project's root using the following commands:
python3 examples/simple_workers.py
python3 -m pso examples/accounts.pso.py
python3 examples/producer_consumer.py

An implementation to rank your favourite songs from World of Walker

World-Of-Walker-Elo An implementation to rank your favourite songs from Alan Walker's 2021 album World of Walker. Uses the Elo rating system, which is

1 Nov 26, 2021
Devil - Very Semple Auto Filter V1 Bot

Devil Very Semple Auto Filter V1 Bot

2 Jun 27, 2022
Pypot ⚙️ A Python library for Dynamixel motor control

Pypot ⚙️ A Python library for Dynamixel motor control Pypot is a cross-platform Python library making it easy and fast to control custom robots based

Poppy Project 238 Nov 21, 2022
An example of python package

An example of python package Why use packages? It is a good practice to not code the same function twice, and to reuse common code from one python scr

10 Oct 18, 2022
A custom advent of code I am completing

advent-of-code-custom A custom advent of code I am doing in python. The link to the problems I am solving is here: https://github.com/seldoncode/Adven

Rocio PV 2 Dec 11, 2021
Python client library for the Databento API

Databento Python Library The Databento Python client library provides access to the Databento API for both live and historical data, from applications

Databento, Inc. 35 Dec 24, 2022
Margin Calculator - Personally tailored investment tool

Margin Calculator - Personally tailored investment tool

1 Jul 19, 2022
An open-source hyper-heuristic framework for multi-objective optimization

MOEA-HH An open-source hyper-heuristic framework for multi-objective optimization. Introduction The multi-objective optimization technique is widely u

Hengzhe Zhang 1 Feb 10, 2022
Explore-bikeshare-data - GitHub project as part of the Programming for Data Science with Python Nanodegree from Udacity

Date created February 10, 2022 Project Title Explore US Bikeshare Data Descripti

Thárcyla 1 Feb 14, 2022
Python 101 Forever

🚀 Python 101 Forever 🚀 Official Python 101 Forever GitHub repository. START HERE - CHECK README SUBSCRIBE FOR UPDATES HERE Sponsors Contac

Hack Bulgaria 58 Nov 30, 2022
2 Way Sync Between Notion Database and Google Calendar

Notion-and-Google-Calendar-2-Way-Sync 2 Way Sync Between a Notion Database and Google Calendar WARNING: This repo will be undergoing a good bit of cha

248 Dec 26, 2022
RangDev Notepad App With Python

RangDev Notepad-App-With-Python Take down quick and speedy notes! This is a small project of a notepad app built with Tkinter and SQLite3. Database cr

rangga.alrasya 1 Dec 01, 2021
LSO, also known as Linux Swap Operator, is a software with both GUI and terminal versions that you can manage the Swap area for Linux operating systems.

LSO - Linux Swap Operator Türkçe - LSO Nedir? LSO, diğer adıyla Linux Swap Operator Linux işletim sistemleri için Swap alanını yönetebileceğiniz hem G

Eren İnce 4 Feb 09, 2022
The next generation Canto RSS daemon

Canto Daemon This is the RSS backend for Canto clients. Canto-curses is the default client at: http://github.com/themoken/canto-curses Requirements De

Jack Miller 155 Dec 28, 2022
Unofficial Python implementation of the DNMF overlapping community detection algorithm

DNMF Unofficial Python implementation of the Discrete Non-negative Matrix Factorization (DNMF) overlapping community detection algorithm Paper Ye, Fan

Andrej Janchevski 3 Nov 30, 2021
Kunai Shitty Raider Leaked LMFAO

Kunai-Raider-Leaked Kunai Shitty Raider Leaked LMFA

5 Nov 24, 2021
A Notifier Program that Notifies you to relax your eyes Every 15 Minutes👀

Every 15 Minutes is an application that is used to Notify you to Relax your eyes Every 15 Minutes, This is fully made with Python and also with the us

FSP Gang s' YT 2 Nov 11, 2021
Pre-commit hook for upgrading type hints

This is a pre-commit hook configured to automatically upgrade your type hints to the new native types implemented in PEP 585.

snok 54 Nov 14, 2022
This is a Blender 2.9 script for importing mixamo Models to Godot-3

Mixamo-To-Godot This is a Blender 2.9 script for importing mixamo Models to Godot-3 The script does the following things Imports the mixamo models fro

8 Sep 02, 2022
Procscan is a quick and dirty python script used to look for potentially dangerous api call patterns in a Procmon PML file.

PROCSCAN Procscan is a quick and dirty python script used to look for potentially dangerous api call patterns in a Procmon PML file. Installation git

Daniel Santos 9 Sep 02, 2022