CPython extension implementing Shared Transactional Memory with native-looking interface

Overview

CPython extension implementing Shared Transactional Memory with native-looking interface

This is my alternative to channels and pickle for cross-interpreter communication for the ongoing PEP 554 -- Multiple Interpreters in the Stdlib and multi-core-python. Treat this project as a deep rework of a standard multiprocessing.sharedctypes, intended to implement a support for complex dynamic data and complex atomic operations.

See INSTALL.txt for instructions on setup. Only 32-bit Linux or Windows are supported, and probably CPython 3.7+. It's still kinda proof-of-concept, so don't expect too much from it.

https://habr.com/en/post/585320/ - Detailed article with examples, benchmarks, and review of implementation details.

Brief tutorial on usage

  1. Import the library using import pso;
  2. Primary process calls pso.init() and notes the returned unique ID of the session;
  3. Secondary processes join the session with pso.connect(ID);
  4. Now all the processes can access a common pso.root() storage via attributes of this object.

Currently the pso.root() shared storage can be used for such types as: str, bytes, bool, int, float, dict, list, tuple, instances of pso.ShmObject and its subclasses, and any combination of mentioned data types.

For example, you can run this code side by side in two terminals:

Primary Secondary
>>> import pso >>> import pso
>>> pso.init()
'pso_FHu0a4idD'
>>> pso.connect('pso_FHu0a4idD')
>>> r = pso.root()
>>> r.list = []
r = pso.root()
>>> r.list
ShmList: 0 elements , empty
>>> r.list.append('stringy')
>>> r.list
ShmList: 1 elements , first element: stringy
>>> r.list
ShmList: 1 elements , first element: stringy

Those are basics you can do with multiprocessing.ShareableList too... except you cannot grow the list after creation and you cannot put other mutable containers into it. But we are just getting started.

Transactions

Extending the two terminals example:

Primary Secondary
>>> pso.transaction_start()
>>> r.list.append('another string')
>>> len(r.list)
1
>>> pso.transaction_commit()
>>> len(r.list)
2
>>> pso.transaction_start()
>>> r.list.append('third string')
>>> r.list
execution pauses
>>> pso.transaction_commit() ShmList: 3 elements , first element: stringy

As you can see, the changes made within transaction are invisible to other processes until committed. Moreover, modified uncommitted objects cannot be read by other processes. By making multiple changes withing a single transaction, you can commit them atomically and be sure nobody sees half-committed inconsistent data.

At first it might look like a simple fine-grained locking, but actually it magically resolves deadlocks and resource starvation. Let's repeat the last operation again but do the reading part with a longer running transaction:

Primary Secondary
>>> pso.transaction_start()
>>> r.list
ShmList: 3 elements , first element: stringy
>>> pso.transaction_start()
>>> r.list.append('fourth string')
Traceback (most recent call last):
File "", line 1, in
pso.ShmAbort: Transaction aborted

Oops, what happenned? The two transactions contended for the same resource again, but the result is different. Obviously, one transaction should be terminated and the other should keep running. PSO knows which one of them is running longer and allows it to proceed, while a recently started transaction is aborted.

In this very example the primary process is left in a limbo state where you are required to handle the contention somehow e.g. do pso.transaction_abort(). However, there already exist functions that can handle everything for you. Those are: pso.transaction() which accepts a function to be executed atomically, and a special with transaction: syntactic sugar which can run an unnamed block atomically e.g.:

with transaction:
    r.list.append('Fifth element')

This syntactic sugar requires a special module loader, activated either by running python3 -m pso modulename.py or by naming a loaded module with a .pso suffix, like modulename.pso.py. The latter option is only available once the "pso" module is loaded, so it won't work for the main module of your project.

You can find example programs in the examples/ folder, those are launched from the project's root using the following commands:
python3 examples/simple_workers.py
python3 -m pso examples/accounts.pso.py
python3 examples/producer_consumer.py

[arXiv 2020] Video Representation Learning with Visual Tempo Consistency

Video Representation Learning with Visual Tempo Consistency [Paper] [Project Page] News Full codebae is coming soon Pretained Models For now, we provi

DeciForce: Crossroads of Machine Perception and Autonomy 24 Nov 23, 2022
Protocol Buffers for the Rest of Us

Protocol Buffers for the Rest of Us Motivation protoletariat has one goal: fixing the broken imports for the Python code generated by protoc. Usage He

Phillip Cloud 76 Jan 04, 2023
An html wrapper for python

MessySoup What is it? MessySoup is a python wrapper for html elements. While still a ways away, the main goal is to be able to build a wesbite straigh

4 Jan 05, 2022
A weekly dive into commonly used modules in the Rust ecosystem, with story flavor!

The goal of this project is to bring the same concept as PyMOTW to the Rust world. PyMOTW was an invaluable resource for me when I was learning Python years ago, and I hope that I can help someone in

Scott Lyons 20 Aug 26, 2022
用于导出墨墨背单词的词库,并生成适用于 List 背单词,不背单词,欧陆词典等的自定义词库

maimemo-export 用于导出墨墨背单词的词库,并生成适用于 List 背单词,欧陆词典,不背单词等的自定义词库。 仓库内已经导出墨墨背单词所有自带词库(暂不包括云词库),多达 900 种词库,可以在仓库中选择需要的词库下载(下载单个文件的方法),也可以去 蓝奏云(密码:666) 下载打包好

ourongxing 293 Dec 29, 2022
🙌Kart of 210+ projects based on machine learning, deep learning, computer vision, natural language processing and all. Show your support by ✨ this repository.

ML-ProjectKart 📌 Repository This kart showcases the finest collection of all projects based on machine learning, deep learning, computer vision, natu

Prathima Kadari 203 Dec 28, 2022
This repository contains the code for the python introduction lab

This repository contains the code for the python introduction lab. The purpose is to have a fairly simple python assignment that introduces the basic features and tools of python

1 Jan 24, 2022
My tools box script for sigma

sigma_python_toolbox My tools box script for sigma purpose My goal is not to replace sigma but to put at disposal the scripts that I think to help me

4 Jun 20, 2022
Visualization of COVID-19 Omicron wave data in Seoul, Osaka, Tokyo, Hong Kong and Shanghai. 首尔、大阪、东京、香港、上海由新冠病毒 Omicron 变异株引起的本轮疫情数据可视化分析。

COVID-19 in East Asian Megacities This repository holds original Python code for processing and visualization COVID-19 data in East Asian megacities a

STONE 10 May 18, 2022
A Regex based linter tool that works for any language and works exclusively with custom linting rules.

renag Documentation Available Here Short for Regex (re) Nag (like "one who complains"). Now also PEGs (Parsing Expression Grammars) compatible with py

Ryan Peach 12 Oct 20, 2022
Design-by-contract in Python3 with informative violation messages and inheritance

icontract icontract provides design-by-contract to Python3 with informative violation messages and inheritance. It also gives a base for a flourishing

275 Jan 02, 2023
little proyect to organize myself, but maybe can help someone else

TaskXT 0.1 Little proyect to organize myself, but maybe can help someone else Idea The main idea is to ogranize you work and stuff to do, but with onl

Gabriel Carmona 4 Oct 03, 2021
SimplePyBLE - Python bindings for SimpleBLE

The ultimate fully-fledged cross-platform Python BLE library, designed for simplicity and ease of use.

Open Bluetooth Toolbox 27 Aug 28, 2022
Gaia: a chrome extension that curates environmental news of a company

Gaia - Gaia: Your Environment News Curator Call for Code 2021 Gaia: a chrome extension that curates environmental news of a company Explore the docs »

4 Mar 19, 2022
Santa's kitchen helper for python

Santa's Kitchen Helper Introduction/Overview Contents UX User Stories Design Wireframes Color Scheme Typography Imagery Features Exisiting Features Fe

Paul Browne 4 May 31, 2022
A Bot that adds YouTube views to your video of choice

YoutubeViews Free Youtube viewer bot A Bot that adds YouTube views to your video of choice Installation git clone https://github.com/davdtheemonk/Yout

ProbablyX 5 Dec 06, 2022
Customizable-menu-python - User customizable menu in Python

Menu personalizável pelo usuário em Python A minha ideia com esse projeto pessoa

Renan Barbosa 4 Oct 28, 2022
This repository contains the exercices for the robotics class at Supaero, 2022.

Supaero robotics, 2022 This repository contains the exercices for the robotics class at Supaero, 2022. The exercices are organized by notebook. Each n

Gepetto team, LAAS-CNRS 5 Aug 01, 2022
Generic NDJSON importer for hashlookup server

Generic NDJSON importer for hashlookup server Usage usage: hashlookup-json-importer.py [-h] [-v] [-s SOURCE] [-p PARENT] [--parent-meta PARENT_META [P

hashlookup 2 Jan 19, 2022
Nateve transpiler developed with python.

Adam Adam is a Nateve Programming Language transpiler developed using Python. Nateve Nateve is a new general domain programming language open source i

Nateve 7 Jan 15, 2022