A python tool for synchronizing the messages from different threads, processes, or hosts.

Related tags

Miscellaneouspythonsynchronizationstreammultiprocessingpython-librarystdoutpython3stdout-redirection
Overview

Sync-stream

This project is designed for providing the synchoronization of the stdout / stderr among different threads, processes, devices or hosts. The package could be used for the following cases:

  1. Use syncstream.LineBuffer: Multiple threads are created. The messages (stdout) from different threads are required to be collected.
  2. Use syncstream.LineProcBuffer in the main process, and syncstream.LineProcMirror in the sub-processes: Multiple sub-processes are created on the same device. The stdout / stderr of each process is redirected to a LineProcMirror, and the results are collected by LineProcBuffer.
  3. Use syncstream.LineFileBuffer: Multiple processes are created. These processes may be deployed on different devices (even with different platforms), but all devices could get accessed to the same shared disk. In this case, the message could be shared by locked files. Each process would hold an independent LineFileBuffer pointing to the same log files.
  4. Use syncstream.LineHostBuffer on the server side, and syncstream.LineHostMirror on the client side: Multiple processes are deployed on different devices, and they could not share the same disk. In this case, the message are synchronized by the web service. Each process would hold a LineHostMirror, and the results would be collected by LineHostBuffer.

The basic package would not contain the file and host modules. To install the package, please use the following options:

pip install syncstream[option1,option2...]
Option Supports
file Install dependencies for the file module. The module provides syncstream.LineFileBuffer.
host Install dependencies for the host module. The module provides syncstream.LineHostBuffer, and syncstream.LineHostMirror.

Documentation

View the documentation here: 📘 https://cainmagi.github.io/sync-stream/

Update reports

0.3.3 @ 6/29/2021

  1. Fix small typos.
  2. Bump the dependencies to the newest versions.

0.3.2 @ 6/14/2021

  1. Fix a bug caused by stopping the mirrors.
  2. Format the meta-data defined in setup.py.
  3. Add the documentation. Currently only the tutorial is finished.

0.3.0 @ 6/4/2021

  1. Support the stop signal for mproc and host modules.
  2. Fix some bugs in the testing script.
  3. Fix typos.

0.2.2 @ 5/25/2021

  1. Add clear() methods for all buffers and mirrors.
  2. Fix typos in the package setup and info file.
  3. Fix a bug caused by writing data to the host in the testing scripts for Linux.

0.2.1 @ 5/24/2021

  1. Add the PyPI publish workflow.

0.2.0 @ 5/24/2021

  1. Finish the synchronization based on the file lock package fasteners.
  2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
  3. Fix the compatibility of the testing scripts for py36, py37.

0.1.0 @ 5/22/2021

  1. Finish the synchronization based on the stdlib.
  2. Create this project.
You might also like...
Provide error messages for Python exceptions, even if the original message is empty

errortext is a Python package to provide error messages for Python exceptions, even if the original message is empty.

Thomas Aglassinger 0 Dec 7, 2021
Telegram bot to remove the forwarded tag from messages.
Telegram bot to remove the forwarded tag from messages.

Anonymous Sender Bot @AnonySendBot Telegram bot to remove the forwarded tag from messages. Table of Contents Usage Deploy To Heroku Local Deploying En

Stark Bots 26 Nov 24, 2022
Automatically remove user join messages when the user leaves the server.

CleanLeave Automatically remove user join messages when the user leaves the server. Installation You will need to install poetry to run this bot local

null 11 Sep 19, 2022
Simple script with AminoLab to send ghost messages

Simple script with AminoLab to send ghost messages

Moleey 1 Nov 22, 2021
A Github Action for sending messages to a Matrix Room.
A Github Action for sending messages to a Matrix Room.

matrix-commit A Github Action for sending messages to a Matrix Room. Screenshot: Example Usage: # .github/workflows/matrix-commit.yml on: push:

null 3 Sep 11, 2022
ViberExport - Export messages from Viber messenger using viber.db file

📲 ViberExport Export messages from Viber messenger using viber.db file ⚡ Usage:

null 7 Nov 23, 2022
Telop - Encode and decode messages using an interpretation of the telegraphic code devised by José María Mathé

telop Telop (TELégrafoÓPtico) - Utilidad para codificar y descodificar mensajes de texto empleando una interpretación del código telegráfico ideado po

Ricardo F. 4 Nov 1, 2022
Automatically unpin old messages so you can always pin more!

PinRotate Automatically unpin old messages so you can always pin more! Installation You will need to install poetry to run this bot locally for develo

null 3 Sep 18, 2022
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool
A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme tool

Oppo/Realme Flash .OFP File on Bootloader A tool to flash .ofp files in bootloader mode without needing MSM Tool, an alternative to official realme to

Italo Almeida 70 Jan 2, 2023
Releases(0.3.3)

  • 0.3.3(Jun 29, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.3 @ 6/29/2021

    1. Fix small typos.
    2. Bump the dependencies to the newest versions.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.2(Jun 14, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.2 @ 6/14/2021

    1. Fix a bug caused by stopping the mirrors.
    2. Format the meta-data defined in setup.py.
    3. Add the documentation. Currently, only the tutorial is finished.
    Source code(tar.gz)
    Source code(zip)

  • 0.3.0(Jun 4, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.3.0 @ 6/4/2021

    1. Support the stop signal for mproc and host modules.
    2. Fix some bugs in the testing script.
    3. Fix typos.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.2(May 25, 2021)

    Introduction

    Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.2 @ 5/25/2021

    1. Add clear() methods for all buffers and mirrors.
    2. Fix typos in the package setup and info file.
    3. Fix a bug caused by writing data to the host in the testing scripts for Linux.
    Source code(tar.gz)
    Source code(zip)

  • 0.2.1(May 24, 2021)

    Introduction

    This is the first release of this project. Sync-stream is a pure python tool for synchronizing the messages (including stdout, stderr, and raised exceptions or warnings) among threads, processes, devices, and hosts. This powerful tool is useful for capturing the updating logs from the sub-processes in some projects.

    Updating reports

    0.2.1 @ 5/24/2021

    1. Add the PyPI publish workflow.

    0.2.0 @ 5/24/2021

    1. Finish the synchronization based on the file lock package fasteners.
    2. Finish the synchronization based on the web service packages flask, flask-restful and urllib3.
    3. Fix the compatibility of the testing scripts for py36, py37.

    0.1.0 @ 5/22/2021

    1. Finish the synchronization based on the stdlib.
    2. Create this project.
    Source code(tar.gz)
    Source code(zip)
Owner
Yuchen Jin
A Ph.D student in University of Houston (UH). Interested area includes: machine learning, programming and religion.
Yuchen Jin
GitHub Repository https://cainmagi.github.io/sync-stream/
A custom advent of code I am completing

advent-of-code-custom A custom advent of code I am doing in python. The link to the problems I am solving is here: https://github.com/seldoncode/Adven

Rocio PV 2 Dec 11, 2021
Subscribe, listen and (in the future) download your favorite podcasts, quickly and easily.

Minimal Podcasts Player https://github.com/son-link/minimal-podcasts-player Subscribe, listen and (in the future) download your favorite podcasts, qui

Alfonso Saavedra 14 Nov 11, 2022
Shell Trality API for local development.

Trality Simulator Intro This package is a work in progress. It allows local development of Trality bots in an IDE such as VS Code. The package provide

CrypTrality 1 Nov 17, 2021
Python MQTT v5.0 async client

gmqtt: Python async MQTT client implementation. Installation The latest stable version is available in the Python Package Index (PyPi) and can be inst

Gurtam 306 Jan 03, 2023
A Lego Mindstorm robot for dealing out cards based on a birds-eye view of a poker table and given ArUco fiducial tags.

A Lego Mindstorm robot for dealing out cards based on a birds-eye view of a poker table and given ArUco fiducial tags.

4 Dec 06, 2021
A new mini-batch framework for optimal transport in deep generative models, deep domain adaptation, approximate Bayesian computation, color transfer, and gradient flow.

BoMb-OT Python3 implementation of the papers On Transportation of Mini-batches: A Hierarchical Approach and Improving Mini-batch Optimal Transport via

Khai Ba Nguyen 18 Nov 14, 2022
Repository specifically for tcss503-22-wi Students

TCSS503: Algorithms and Problem Solving for Software Developers Course Description Introduces advanced data structures and key algorithmic techniques

Kevin E. Anderson 3 Nov 08, 2022
A python package to adjust the bias of probabilistic forecasts/hindcasts using "Mean and Variance Adjustment" method.

Documentation A python package to adjust the bias of probabilistic forecasts/hindcasts using "Mean and Variance Adjustment" method. Read documentation

1 Feb 02, 2022
Brython (Browser Python) is an implementation of Python 3 running in the browser

brython Brython (Browser Python) is an implementation of Python 3 running in the browser, with an interface to the DOM elements and events. Here is a

5.9k Jan 02, 2023
DNA Storage Simulator that analyzes and simulates DNA storage

DNA Storage Simulator This monorepository contains code for a research project by Mayank Keoliya and supervised by Djordje Jevdjic, that analyzes and

Mayank Keoliya 3 Sep 25, 2022
A GUI love Calculator which saves all the User Data in text file(sql based script will be uploaded soon). Interative GUI. Even For Admin Panel

Love-Calculator A GUI love Calculator which saves all the User Data in text file(sql based script will be uploaded soon). Interative GUI, even For Adm

Adithya Krishnan 1 Mar 22, 2022
Fetch data from an excel file and create HTML file

excel-to-html Problem Statement! - Fetch data from excel file and create html file Excel.xlsx file contain the information.in multiple rows that is ne

Vivek Kashyap 1 Oct 25, 2021
Fisherman is a free open source fishing bot written in python.

Fisherman is a free open source fishing bot written in python.

Pure | Cody 33 Jan 29, 2022
List of all D&D 5e monsters: WotC + popular third-party sourcebooks

Xio's Guide to Monsters If you're a DM like me, and you have multiple sources of D&D 5e monsters that include WotC as well as third-party suppliers, y

20 Jan 06, 2023
It is convenient to quickly import Python packages from the network.

It is convenient to quickly import Python packages from the network.

zmaplex 1 Jan 18, 2022
Python with braces. Because Python is awesome, but whitespace is awful.

Bython Python with braces. Because Python is awesome, but whitespace is awful. Bython is a Python preprosessor which translates curly brackets into in

1 Nov 04, 2021
A simple method to create strong password.

A simple method to create strong password.

1 Jan 23, 2022
A web project to control the daily life budget planing

Budget Planning - API In this repo there's only the API and Back-End of the this project. Install and run the project # install virtualenv --python=py

Leonardo Da Vinci 1 Oct 24, 2021
Sequence clustering and database creation using mmseqs, from local fasta files

Sequence clustering and database creation using mmseqs, from local fasta files

Ana Julia Velez Rueda 3 Oct 27, 2022
Python script for diving image data to train test and val

dataset-division-to-train-val-test-python python script for dividing image data to train test and val If you have an image dataset in the following st

Muhammad Zeeshan 1 Nov 14, 2022