Lightweight library for accessing data and configuration

Related tags

Miscellaneousaccsr
Overview

accsr

This lightweight library contains utilities for managing, loading, uploading, opening and generally wrangling data and configurations. It was battle tested in multiple projects at appliedAI.

Please open new issues for bugs, feature requests and extensions. See more details about the structure and workflow in the developer's readme.

Overview

Source code documentation and usage examples are here. We also provide notebooks with examples in TODO.

Installation

Install the latest release with

pip install accsr

To live on the edge, install the latest develop version with

pip install --pre accsr
Comments
  • Dryrun-based pull/push and tqdm

    Dryrun-based pull/push and tqdm

    Added tqdm progress bar to RemoteStorage.push/pull methods

    • First determines the total number of bytes to push/pull
    • Updates the number of transferred bytes after every file

    resolve #4 #5 #9

    opened by fariedabuzaid 4
  • Fix test failure because of failed name resolution

    Fix test failure because of failed name resolution

    This PR fixes the test failure related to failed name resolution on CI. It makes the tests run inside a container because this is apparently needed to enable host name resolution for the minio service from within the tests.

    You can refer to these links for more information:

    • https://docs.github.com/en/actions/using-containerized-services/about-service-containers#mapping-docker-host-and-service-container-ports
    • https://docs.github.com/en/actions/using-containerized-services/creating-redis-service-containers#running-jobs-in-containers
    opened by AnesBenmerzoug 2
  • Add Simulation Mode

    Add Simulation Mode

    Add a new flag to the RemoteStorage push/pull operation. If True the function should determine and return the operations that need to be conducted without actually performing them.

    enhancement 
    opened by fariedabuzaid 0
  • Transactional safety for push and pull in remote storage

    Transactional safety for push and pull in remote storage

    Currently, pushing and pulling of directories does not check whether the entire operation can be performed successfully (e.g. if modified files already exist and overwrite_existing=False). This leads to a partial execution before an error is thrown and thus to an unpredictable state.

    We should check if the entire operation can be performed before pushing/pulling anything.

    Also, to be more familiar to git users, overwrite_existing should be renamed to force. This is a breaking change, the minor version should be bumped

    enhancement 
    opened by MischaPanch 0
  • chore: release version 0.3.5-dev0

    chore: release version 0.3.5-dev0

    @MischaPanch can you release the current dev branch? I found a bug in the old version which seems to fixed now. Would be great to get the fix installed. Not urgent though, I can work with the dev branch for now :)

    opened by slettner 0
  • Improve docs by extending notebooks

    Improve docs by extending notebooks

    We have essentially no documentation on how to use accsr. The interplay of storage and config modules should be demonstrated in notebooks. See tests/conftest.py for an example how a storage service is instantiated during local testing an in CI.

    documentation 
    opened by MischaPanch 0
  • Move convenient path selections in push/pull

    Move convenient path selections in push/pull

    We should make it easier to push/pull a bunch of paths based on patterns. For that we should add

    • [ ] Permit passing glob-patterns to push/pull
    • [ ] Add the possibility to pass a regex as except_matches kwarg to permit simple exclusion of files. The current regex kwarg should be renamed to if_matches.

    This would permit things like

    storage.push("data/**/*.jpg", except_matches=r".*test.*") 
    

    We could additionally allow passing a except_condition: Callable[[str]], bool] = None (or do you think if_condition is more natural?), in which case the above can be rewritten

    storage.push("data/**/*.jpg", except_condition=lambda n: "test" in n) 
    

    The condition could be even made more general, mapping the metadata-object to a bool (thereby e.g. allowing filtering by size), at the cost of a more complicated interface for callables. @fariedabuzaid @AnesBenmerzoug what do you think?

    enhancement 
    opened by MischaPanch 0
  • CI: make caching work within containers

    CI: make caching work within containers

    @MischaPanch This fixes the tests but it currently break caching.

    WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
    

    I think running jobs inside containers is the way to go and we should invest some time to make caching work with it.

    Originally posted by @AnesBenmerzoug in https://github.com/appliedAI-Initiative/accsr/issues/1#issuecomment-976536009

    Build/CI 
    opened by MischaPanch 0
Releases(v0.3.4)
Owner
appliedAI Initiative
The appliedAI Initiative aims to lift Germany and Europe to the AI age by accelerating the adoption of AI technology
appliedAI Initiative
💡 Fully automatic light management based on conditions like motion, illuminance, humidity, and other clever features

Fully automatic light management based on motion as AppDaemon app. 🕓 multiple daytimes to define different scenes for morning, noon, ... 💡 supports

Ben 105 Dec 23, 2022
Simple card retirement plugin for Anki

Anki Retirement Addon Allow users to suspend, tag, delete, or move cards that reach a specific retirement interval Supports Anki version 2.1.45 Licens

3 Dec 23, 2022
Provide error messages for Python exceptions, even if the original message is empty

errortext is a Python package to provide error messages for Python exceptions, even if the original message is empty.

Thomas Aglassinger 0 Dec 07, 2021
Script em python, utilizando PySimpleGUI, para a geração de arquivo txt a ser importado no sistema de Bilhetagem Eletrônica da RioCard, no Estado do Rio de Janeiro.

pedido-vt-riocard Script em python, utilizando PySimpleGUI, para a geração de arquivo txt a ser importado no sistema de Bilhetagem Eletrônica da RioCa

Carlos Bruno Gomes 1 Dec 01, 2021
Software that extracts spreadsheets from various .pdf files to .csv

Extração de planilhas de diversos arquivos .pdf para .csv O código inteiro foi desenvolvido em Python. Foi utilizado o pacote "tabula" e a biblioteca

Marcos Silva 2 Jan 09, 2022
Research on how Gboard Stickers work.

Google-Sticker-Mashup-Research Research on how Gboard Stickers work. Contribute Contributing is nice, and you will be listed below for contributing. C

Jeremiah 45 Oct 28, 2022
An end-to-end encrypted chat

An end-to-end encrypted chat, that allows users to anonymously talk without ip logs, personal info, or need for registration.

Privalise 1 Nov 27, 2021
A slapdash script to solve Wordle or Absurdle automatically

A slapdash script to solve Wordle or Absurdle automatically

Michael Anthony 1 Jan 19, 2022
program to store and update pokemons using SQL and Flask

Pokemon SQL and Flask Pokemons api in python. Technologies flask pymysql Description PokeCorp is a company that tracks pokemon and their trainers arou

Sara Hindy Salfer 1 Oct 20, 2021
Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

6 Sep 23, 2022
原神抽卡记录导出

原神抽卡记录导出 抽卡记录分析工具 from @笑沐泽 抽卡记录导出工具js版,含油猴脚本可在浏览器导出 注意:我的是python版,带饼图的是隔壁electron版,功能类似 Wik

834 Jan 04, 2023
Job Guy Backend

جاب‌گای چیست؟ اونجا وضعیت چطوریه؟ یه سوال به همین کلیت و ابهام معمولا وقتی برای یه شرکت رزومه می‌فرستیم این سوال کلی و بزرگ برای همه پیش میاد.اونجا وض

Jobguy.work 217 Dec 25, 2022
Grouping nucleotide coordinate ranges.

NuclRanger Grouping nucleotide coordinate ranges. A quick pre-processing step for "bedtools getfasta":- https://bedtools.readthedocs.io/en/latest/cont

Sujanavan Tiruvayipati 1 Oct 04, 2022
Cisco IOS-XE Operations Program. Shows operational data using restconf and yang

XE-Ops View operational and config data from devices running Cisco IOS-XE software. NoteS The build folder is the latest build. All other files are fo

18 Jul 23, 2022
Snakemake worflow to process and filter long read data from Oxford Nanopore Technologies.

Nanopore-Workflow Snakemake workflow to process and filter long read data from Oxford Nanopore Technologies. It is designed to compare whole human gen

5 May 13, 2022
Example applications, dashboards, scripts, notebooks, and other utilities built using Polygon.io

Polygon.io Examples Example applications, dashboards, scripts, notebooks, and other utilities built using Polygon.io. Examples Preview Name Type Langu

Tim Paine 4 Jun 01, 2022
Generate Gaussian 09 input files for the rotamers of an input compound.

Rotapy Purpose Generate Gaussian 09 input files for the rotamers of an input compound. Distance to the axis of rotation remains constant throughout th

1 Jul 16, 2021
Projects and assets from Wireframe #56

Wireframe56 Projects and assets from Wireframe #56 Make a Boulder Dash level editor in Python, pages 50-57, by Mark Vanstone. Code an homage to Bubble

Wireframe magazine 10 Sep 07, 2022
Simple calculator with random number button and dark gray theme created with PyQt6

Calculator Application Simple calculator with random number button and dark gray theme created with : PyQt6 Python 3.9.7 you can download the dark gra

Flamingo 2 Mar 07, 2022
A python script for practicing Toki Pona.

toki.py A python script for practicing Toki Pona. Modified from a hirigana script by ~vilmibm. Example of the script running: $ ./toki.py This script

Dustin 2 Dec 09, 2021