Python package for reference counting native pointers

Overview

refcount

license status Documentation Status master: Build status - master testing: Build status - devel

Reference counted native handles

This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boils down to "simply" maintaining a set of counters, it is deceptively complicated to do so properly and not end up with memory leaks or crashes. This package offers structured options for reliably managing external native resources. Surprisingly I could not locate an existing package doing just what I needed. Other use cases requiring reference counting, aside from native library resources, may benefit from reusing and extending classes in refcount.

refcount from version 0.7 includes classes using cffi. Other low-level interoperability mechanisms may well be added in the future.

License

MIT (see License.txt)

Documentation

Hosted at pyrefcount.readthedocs.io

Source code

The code repository is on GitHub.

Installation

pip install refcount

From source:

pip install -r requirements.txt
python setup.py install

Sample use

The following example is based on one of the unit tests.

Say we have a C++ library with objects and a C API:

#define TEST_DOG_PTR  testnative::dog*
#define TEST_OWNER_PTR  testnative::owner*
#define TEST_COUNTED_PTR  testnative::reference_counter*

testnative::dog* create_dog();
testnative::owner* create_owner(testnative::dog* d);
void say_walk(testnative::owner* owner);
void release(testnative::reference_counter* obj);
// etc.

From the outside of the library the API is exported with opaque pointers void* (C structs pointers and native C99 types could be handled too).

void* create_dog();
void* create_owner(void* d);
void say_walk(void* owner);
void release(void* obj);
// etc.

Starting with the end in mind, from Python we want an API hiding the low level details close to the C API, in particular avoiding managing native memory via release C API calls, piggybacking the python GC instead.

dog = Dog()
owner = DogOwner(dog)
owner.say_walk()
print(dog.position)
dog = None # the "native dog" is still alive though, as the owner incremented the ref count
owner = None

This is doable with refcount and the cffi package. One possible design is:

ut_ffi = cffi.FFI()

ut_ffi.cdef('extern void* create_dog();')
ut_ffi.cdef('extern void* create_owner( void* d);')
ut_ffi.cdef('extern void say_walk( void* owner);')
ut_ffi.cdef('extern void release( void* obj);')
# etc.

ut_dll = ut_ffi.dlopen('c:/path/to/test_native_library.dll', 1) # Lazy loading

class CustomCffiNativeHandle(CffiNativeHandle):
    def __init__(self, pointer, prior_ref_count = 0):
        super(CustomCffiNativeHandle, self).__init__(pointer, type_id='', prior_ref_count = prior_ref_count)

    def _release_handle(self) -> bool:
        ut_dll.release(self.get_handle())
        return True

class Dog(CustomCffiNativeHandle):
    def __init__(self, pointer = None):
        if pointer is None:
            pointer = ut_dll.create_dog()
        super(Dog, self).__init__(pointer)
    # etc.

class DogOwner(CustomCffiNativeHandle):

    def __init__(self, dog):
        super(DogOwner, self).__init__(None)
        self._set_handle(ut_dll.create_owner(dog.get_handle()))
        self.dog = dog
        self.dog.add_ref() # Do note this important reference increment

    def say_walk(self):
        ut_dll.say_walk(self.get_handle())

    def _release_handle(self) -> bool:
        super(DogOwner, self)._release_handle()
        # super(DogOwner, self)._release_handle()
        self.dog.release()
        return True

Related work

Ancestry, acknowledgements

This python package refcount actually spawned from prior work for interoperability between C++, R and .NET (R.NET)

refcount features using cffi were also significantly informed by Kevin Plastow's work while he was at the Australian Bureau of Meteorology; this contribution is gratefully acknowledged.

In you have native interop needs you may also want to look at:

Other python packages

refcount was created in part because no existing prior (Python) work could quite fit the need. There are however packages that may better address your particular need:

You might also like...
A test repository to build a python package and publish the package to Artifact Registry using GCB

A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.

PSP (Python Starter Package) is meant for those who want to start coding in python but are new to the coding scene.

Python Starter Package PSP (Python Starter Package) is meant for those who want to start coding in python, but are new to the coding scene. We include

PyPIContents is an application that generates a Module Index from the Python Package Index (PyPI) and also from various versions of the Python Standard Library.

PyPIContents is an application that generates a Module Index from the Python Package Index (PyPI) and also from various versions of the Python Standar

Example python package with pybind11 cpp extension

Developing C++ extension in Python using pybind11 This is a summary of the commands used in the tutorial.

qecsim is a Python 3 package for simulating quantum error correction using stabilizer codes.

qecsim qecsim is a Python 3 package for simulating quantum error correction using stabilizer codes.

A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.
A command-line utility that creates projects from cookiecutters (project templates), e.g. Python package projects, VueJS projects.

Cookiecutter A command-line utility that creates projects from cookiecutters (project templates), e.g. creating a Python package project from a Python

A toy repo illustrating a minimal installable Python package

MyToy: a minimal Python package This repository contains a minimal, toy Python package with a few files as illustration for students of how to lay out

Python package that mirrors the original Nodejs ReplAPI-It.

Python-ReplAPI-It Python package that mirrors the original Nodejs ReplAPI-It. Contributing First fork the repo: $ git clone https://github.com/ReplAPI

This project recreates the R-based RCy3 Cytoscape Automation library as a Python package.

Python library for calling Cytoscape Automation via CyREST

Comments
  • Roadmap for reference counting around pybind11

    Roadmap for reference counting around pybind11

    opened by jmp75 1
  • CFFI wrappers helper functions for use in codegen

    CFFI wrappers helper functions for use in codegen

    Basically, need to write the equivalent of wrapping functions in an the R package cinterop. While a cinterop Python package will be warranted, the refcount wrapping/unwrapping helper functions should probably live in refcount.

    enhancement 
    opened by jmp75 1
  • Bump certifi from 2022.6.15 to 2022.12.7

    Bump certifi from 2022.6.15 to 2022.12.7

    Bumps certifi from 2022.6.15 to 2022.12.7.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
Releases(v1.1.0)
  • v1.1.0(Aug 19, 2022)

    Main changes

    • Expand some features to cater for macos
    • Tidy up and reengineer some of the legacy functions in the platform utilities putils to facilitate library loading. Minor breaking changes, but probably for no-one but the author in effect.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Aug 13, 2022)

    Main changes

    • Improve documentation, and use the mkdocs-material theme
    • mkdocs.yml changes. Enable dark/light modes from mkdocs-material
    • Improved type hints, and add static type checks (mypy)
    • Improve unit tests and add unit test code coverage reporting
    • Manage the package using poetry; phase out setuptools.
    • Apply some of the approaches in https://py-pkgs.org.
    • Restore appveyor CI to a working state
    Source code(tar.gz)
    Source code(zip)
  • 0.9.3(Aug 13, 2022)

Owner
CSIRO Hydroinformatics
CSIRO - hydroinformatics repositories
CSIRO Hydroinformatics
Model Quantization Benchmark

MQBench Update V0.0.2 Fix academic prepare setting. More deployable prepare process. Fix setup.py. Fix deploy on SNPE. Fix convert_deploy bug. Add Qua

500 Jan 06, 2023
Convert .1pux to .csv

1PasswordConverter Convert .1pux to .csv 1Password uses this new export format .1pux, I assume stands for 1 Password User eXport. As of right now, 1Pa

Shayne Hartford 7 Dec 16, 2022
an elegant datasets factory

rawbuilder an elegant datasets factory Free software: MIT license Documentation: https://rawbuilder.readthedocs.io. Features Schema oriented datasets

Mina Farag 7 Nov 12, 2022
WriteAIr is a website which allows users to stream their writing.

WriteAIr is a website which allows users to stream their writing. It uses HSV masking to detect a pen which the user writes with. Plus, users can select a wide range of options through hand gestures!

Atharva Patil 1 Nov 01, 2021
Demo content - Automate your automation!

Automate-AAP2 Demo Content - Automate your automation! A fully automated Ansible Automation Platform. Context Installing and configuring Ansible Autom

0 Oct 27, 2022
You can easily send campaigns, e-marketing have actually account using cash will thank you for using our tools, and you can support our Vodafone Cash +201090788026

*** Welcome User Sorry I Mean Hello Brother ✓ Devolper and Design : Mokhtar Abdelkreem ========================================== You Can Follow Us O

Mo Code 1 Nov 03, 2021
Excel cell checker with python

excel-cell-checker Description This tool checks a given .xlsx file has the struc

Paul Aumann 1 Jan 04, 2022
A pomodoro app written in Python

Pomodoro It's a pomodoro app written in Python. You can minimize it while you're working if you want to, it'll pop up on your screen when the timer is

Yiğit 1 Dec 20, 2021
A repository for all ZenML projects that are specific production use-cases.

ZenFiles Original Image source: https://www.goodfon.com/wallpaper/x-files-sekretnye-materialy.html And naturally, all credits to the awesome X-Files s

ZenML 66 Jan 06, 2023
Jogo em redes similar ao clássico pedra papel e tesoura

Batalha Tática Tecnologias de Redes de Computadores-A-N-JOGOS DIGITAIS Professor Fabio Henrique Cabrini Alunos: Eric Henrique de Oliveira Silva - RA 1

Eric Henrique de Oliveira Silva 1 Dec 01, 2021
Simulation-Based Inference Benchmark

This repository contains a simulation-based inference benchmark framework, sbibm, which we describe in the associated manuscript "Benchmarking Simulation-based Inference".

SBI Benchmark 58 Oct 13, 2022
Simple Python tool to check if there is an Office 365 instance linked to a domain.

o365chk.py Simple Python script to check if there is an Office365 instance linked to a particular domain.

Steven Harris 37 Jan 02, 2023
Run Python code right in your Telegram messages

Run Python code right in your Telegram messages Made with Telethon library, TGPy is a tool for evaluating expressions and Telegram API scripts. Instal

29 Nov 22, 2022
A python API act as Control Center to control your Clevo Laptop via wmi on windows.

ClevoPyControlCenter A python API act as Control Center to control your Clevo Laptop via wmi on windows. Usage # pip3 install pymi from clevo_wmi impo

3 Sep 19, 2022
A complete python calculator with 2 modes Float and Int numbers.

Python Calculator This program is made for learning purpose. Getting started This Program runs using python, install it via terminal or from thier ofi

Felix Sanchez 1 Jan 18, 2022
Utility to play with ADCS, allows to request tickets and collect information about related objects

certi Utility to play with ADCS, allows to request tickets and collect information about related objects. Basically, it's the impacket copy of Certify

Eloy 185 Dec 29, 2022
Necst-lib - Pure Python tools for NECST

necst-lib Pure Python tools for NECST. Features This library provides: something

NANTEN2 Group 5 Dec 15, 2022
Collection of Python scripts to perform Eikonal Tomography

Collection of Python scripts to perform Eikonal Tomography

Emanuel Kästle 10 Nov 04, 2022
Aevsploit İçin Destekde Bulun Papara: 1427113016

Aevsploit İçin Destekde Bulun Papara: 1427113016 Toolu Geliştirmek İçin Fikirlerinizi Bekliyorum Telegram

9 Jun 07, 2022
Voldemort's Python import helper

importmagician Voldemort's Python import helper pip install importmagician Import from uninstalled Python directories Say you have a directory (relat

Zhengyang Feng 4 Mar 09, 2022