Lightweight library for accessing data and configuration

Related tags

Miscellaneousaccsr
Overview

accsr

This lightweight library contains utilities for managing, loading, uploading, opening and generally wrangling data and configurations. It was battle tested in multiple projects at appliedAI.

Please open new issues for bugs, feature requests and extensions. See more details about the structure and workflow in the developer's readme.

Overview

Source code documentation and usage examples are here. We also provide notebooks with examples in TODO.

Installation

Install the latest release with

pip install accsr

To live on the edge, install the latest develop version with

pip install --pre accsr
Comments
  • Dryrun-based pull/push and tqdm

    Dryrun-based pull/push and tqdm

    Added tqdm progress bar to RemoteStorage.push/pull methods

    • First determines the total number of bytes to push/pull
    • Updates the number of transferred bytes after every file

    resolve #4 #5 #9

    opened by fariedabuzaid 4
  • Fix test failure because of failed name resolution

    Fix test failure because of failed name resolution

    This PR fixes the test failure related to failed name resolution on CI. It makes the tests run inside a container because this is apparently needed to enable host name resolution for the minio service from within the tests.

    You can refer to these links for more information:

    • https://docs.github.com/en/actions/using-containerized-services/about-service-containers#mapping-docker-host-and-service-container-ports
    • https://docs.github.com/en/actions/using-containerized-services/creating-redis-service-containers#running-jobs-in-containers
    opened by AnesBenmerzoug 2
  • Add Simulation Mode

    Add Simulation Mode

    Add a new flag to the RemoteStorage push/pull operation. If True the function should determine and return the operations that need to be conducted without actually performing them.

    enhancement 
    opened by fariedabuzaid 0
  • Transactional safety for push and pull in remote storage

    Transactional safety for push and pull in remote storage

    Currently, pushing and pulling of directories does not check whether the entire operation can be performed successfully (e.g. if modified files already exist and overwrite_existing=False). This leads to a partial execution before an error is thrown and thus to an unpredictable state.

    We should check if the entire operation can be performed before pushing/pulling anything.

    Also, to be more familiar to git users, overwrite_existing should be renamed to force. This is a breaking change, the minor version should be bumped

    enhancement 
    opened by MischaPanch 0
  • chore: release version 0.3.5-dev0

    chore: release version 0.3.5-dev0

    @MischaPanch can you release the current dev branch? I found a bug in the old version which seems to fixed now. Would be great to get the fix installed. Not urgent though, I can work with the dev branch for now :)

    opened by slettner 0
  • Improve docs by extending notebooks

    Improve docs by extending notebooks

    We have essentially no documentation on how to use accsr. The interplay of storage and config modules should be demonstrated in notebooks. See tests/conftest.py for an example how a storage service is instantiated during local testing an in CI.

    documentation 
    opened by MischaPanch 0
  • Move convenient path selections in push/pull

    Move convenient path selections in push/pull

    We should make it easier to push/pull a bunch of paths based on patterns. For that we should add

    • [ ] Permit passing glob-patterns to push/pull
    • [ ] Add the possibility to pass a regex as except_matches kwarg to permit simple exclusion of files. The current regex kwarg should be renamed to if_matches.

    This would permit things like

    storage.push("data/**/*.jpg", except_matches=r".*test.*") 
    

    We could additionally allow passing a except_condition: Callable[[str]], bool] = None (or do you think if_condition is more natural?), in which case the above can be rewritten

    storage.push("data/**/*.jpg", except_condition=lambda n: "test" in n) 
    

    The condition could be even made more general, mapping the metadata-object to a bool (thereby e.g. allowing filtering by size), at the cost of a more complicated interface for callables. @fariedabuzaid @AnesBenmerzoug what do you think?

    enhancement 
    opened by MischaPanch 0
  • CI: make caching work within containers

    CI: make caching work within containers

    @MischaPanch This fixes the tests but it currently break caching.

    WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
    

    I think running jobs inside containers is the way to go and we should invest some time to make caching work with it.

    Originally posted by @AnesBenmerzoug in https://github.com/appliedAI-Initiative/accsr/issues/1#issuecomment-976536009

    Build/CI 
    opened by MischaPanch 0
Releases(v0.3.4)
Owner
appliedAI Initiative
The appliedAI Initiative aims to lift Germany and Europe to the AI age by accelerating the adoption of AI technology
appliedAI Initiative
A small script I made that takes any standard Decklist of magic the gathering cards and pulls all card images from scryfall at once!

A small script I made that takes any standard Decklist of magic the gathering cards and pulls all card images from scryfall at once!

15 Aug 26, 2022
Flames Calculater App used to calculate flames status between two names created using python's Flask web framework.

Flames Finder Web App Flames Calculater App used to calculate flames status between two names created using python's Flask web framework. First, App g

Siva Prakash 4 Jan 02, 2022
Old versions of Deadcord that are problematic or used as reference.

⚠️ Unmaintained and broken. We have decided to release the old version of Deadcord before our v1.0 rewrite. (which will be equiped with much more feat

Galaxzy 1 Feb 10, 2022
Python3 Interface to numa Linux library

py-libnuma is python3 interface to numa Linux library so that you can set task affinity and memory affinity in python level for your process which can help you to improve your code's performence.

Dalong 13 Nov 10, 2022
A Kodi add-on for watching content hosted on PeerTube.

A Kodi add-on for watching content hosted on PeerTube. This add-on is under development so only basic features work, and you're welcome to improve it.

1 Dec 18, 2021
Placeholders is a single-unit storage solution for your Frontend.

Placeholder Placeholders is a single-unit file storage solution for your Frontend. Why Placeholder? Generally, when a website/service requests for fil

Tanmoy Sen Gupta 1 Nov 09, 2021
The git for the Python Story Utility Package library.

PSUP, The Python Story Utility Package Module. PSUP helps making stories or games with options, diverging paths, different endings and so on. You can

Enoki 6 Nov 27, 2022
Using Python to parse through email logs received through several backup systems.

outlook-automated-backup-control Backup monitoring on a mailbox: In this mailbox there will be backup logs. The identification will based on the follo

Connor 2 Sep 28, 2022
Rates how pog a word or user is. Not random and does have *some* kind of algorithm to it.

PogRater :D Rates how pogchamp a word is :D A fun project coded by JBYT27 using Python3 Have you ever wondered how pog a word is? Well, congrats, you

an aspirin 2 Jun 25, 2022
WordlistPasswordGenerator - Shuhfab Basheer

WordlistPasswordGenerator - Shuhfab Basheer Python wordlist generator MAINTAINER

1 Dec 31, 2021
Intelligent Employer Profiling Platform.

Intelligent Employer Profiling Platform Setup Instructions Generating Model Data Ensure that Python 3.9+ and pip is installed. Install project depende

Harvey Donnelly 2 Jan 09, 2022
PythonCalculator - A simple Calculator made in python using tkinter for GUI

PythonCalculator A simple Calculator made in python using tkinter for GUI. For P

ʀᴇxɪɴᴀᴢᴏʀ 1 Jan 01, 2022
Is a util for xferring skinning from one mesh to another

maya_pythonplugins skinTo: Is a util for xferring skinning from one mesh to another args: :param maxInfluences: is the number of max influences on the

James Dunlop 2 Jan 24, 2022
Python Projects is an Open Source to enhance your python skills

Welcome! 👋🏽 Python Project is Open Source to enhance your python skills. You're free to contribute. 🤓 You just need to give us your scripts written

Tristán 6 Nov 28, 2022
A python script providing an idea of how a MindSphere application, e.g., a dashboard, can be displayed around the clock without the need of manual re-authentication on enforced session expiration

A python script providing an idea of how a MindSphere application, e.g., a dashboard, can be displayed around the clock without the need of manual re-authentication on enforced session expiration

MindSphere 3 Jun 03, 2022
Nag0mi ctf problem 2021 writeup

Nag0mi ctf problem 2021 writeup

3 Apr 04, 2022
Run CodeServer on Google Colab using Inlets in less than 60 secs using your own domain.

Inlets Colab Run CodeServer on Colab using Inlets in less than 60 secs using your own domain. Features Optimized for Inlets/InletsPro Use your own Cus

2 Dec 30, 2021
TMTC Commander Core

This commander application was first developed by KSat for the SOURCE project to test the on-board software but has evolved into a more generic tool for satellite developers to perform TMTC (Telemetr

robamu 8 Dec 14, 2022
An OrpheusDL Tidal module

OrpheusDL - Tidal A Tidal module for the OrpheusDL modular archival music program Report Bug · Request Feature Table of content About OrpheusDL - Tida

Daniel 54 Dec 29, 2022
Small C-like language compiler for the Uxn assembly language

Pyuxncle is a single-pass compiler for a small subset of C (albeit without the std library). This compiler targets Uxntal, the assembly language of the Uxn virtual computer. The output Uxntal is not

CPunch 13 Jun 28, 2022