Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
A collection of scripts to steal BTC from Lightning Network enabled custodial services. Only for educational purpose! Share your findings only when design flaws are fixed.

Lightning Network Fee Siphoning Attack LN-fee-siphoning is a collection of scripts to subtract BTC from Lightning Network enabled custodial services b

Reckless_Satoshi 14 Oct 15, 2022
The community bot for the Python Discord community

Python Utility Bot This project is a Discord bot specifically for use with the Python Discord server. It provides numerous utilities and other tools t

Python Discord 998 Jan 03, 2023
A simple discord bot written in python which can surf subreddits, send a random meme, jokes and also weather of a given place

A simple Discord Bot A simple discord bot written in python which can surf subreddits, send a random meme, jokes and also weather of a given place. We

1 Jan 24, 2022
Diablo II Resurrected helper

Diablo II Resurrected 快捷施法辅助 功能: + 创建守护进程,注册全局热键 alt+/ 启用和关闭功能 (todo: 播放声音提示) + 按 x 强制移动 + 按 1 ~ 0 快捷施法到鼠标区域 使用 编辑配置 settings.py 技能信息做如下定义: SKILLS:

Wan 2 Nov 06, 2022
A Telegram Music Bot with proper functions written in Python with Pyrogram and Py-Tgcalls.

⭐️ Yukki Music Bot ⭐️ A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls Ready to use method A Support Group and ready-to-use runnin

Shikhar Kumar 1000 Jan 03, 2023
A Telegram bot for Download songs in mp3 format from YouTube and Extract lyrics from Genius.com ❤️

MeudsaMusic A Telegram bot for Download songs in mp3 format from YouTube and Extract lyrics from Genius.com ❤️ Commands Reach @MedusaMusic on Telegram

Bibee 14 Oct 06, 2022
Python 3 SDK/Wrapper for Huobi Crypto Exchange Api

This packages intents to be an idiomatic PythonApi wrapper for https://www.huobi.com/ Huobi Api Doc: https://huobiapi.github.io/docs Showcase TODO Con

3 Jul 28, 2022
Download videos from Youtube and other platforms through a Telegram Bot

ytdl-bot Download videos from YouTube and other platforms through a Telegram Bot Usage: https://t.me/benny_ytdlbot Send link from YouTube directly to

Telegram Bot Collection 289 Jan 03, 2023
IACR Events Scraper

IACR Events Scraper This scrapes https://iacr.org/events/ and exports it as a calendar file. I host a version of this for myself under https://arrrr.c

Karolin Varner 6 May 28, 2022
Connect your Nintendo Switch playing status to Discord!

Disclaimer: Unfortunately, it appears that Nintendo has removed returning self-Presence in their API as of recently, making this project near obsolete

Deltaion Lee 145 Dec 30, 2022
This is a Telegram video compress bot repo. By Binary Tech💫

This is a Telegram Video Compress Bot. Prouduct By Binary Tech 💫 Features Compresse videos and generate screenshots too.You can set custom video name

silentz lk 2 Jan 06, 2022
fair-test is a library to build and deploy FAIR metrics tests APIs supporting the specifications used by the FAIRMetrics working group.

☑️ FAIR test fair-test is a library to build and deploy FAIR metrics tests APIs supporting the specifications used by the FAIRMetrics working group. I

Maastricht University IDS 6 Oct 30, 2022
A customizable, multilanguage Telegram shop bot with Telegram Payments support

Greed A customizable, multilanguage Telegram shop bot with Telegram Payments support! Demo Send a message to @greedtestbot on Telegram to view a demo

Stefano Pigozzi 328 Dec 29, 2022
A fast, easy to set up telegram userbot running Python 3 which uses fork of the Telethon Library.

forked from friendly-telegram/friendly-telegram Friendly Telegram Userbot A fast, easy to set up telegram userbot running Python 3 which uses fork of

GeekTG 75 Jan 04, 2023
A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls

A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls. This is Also The Source Code of The UserBot Which is Playing Music in @S1-BOTS Support Group ❤️

SAF ONE 224 Dec 28, 2022
Telegram bot for downloading covid-19 vaccine certificate

cowin-certificate-bot This is the source code of @cowincertbot, A telegram bot inspired by the whatsapp bot implementation of indian government for co

ArUn Pt 30 Oct 07, 2022
Python API wrapper library for Convex Value API

convex-value-python Python API wrapper library for Convex Value API. Further Links: Convex Value homepage @ConvexValue on Twitter JB on Twitter Authen

Aaron DeVera 2 May 11, 2022
A Simple Google Translate Bot By VndGroup ❤️ Made With Python

VndGroup Google Translator Heroku Deploy ❤️ Functions This Bot Can Translate 95 Languages We Can Set Custom Language Group Support Mandatory Vars [+]

Venuja Sadew 1 Oct 09, 2022
Repositório para a Live Coding do dia 22/12/2021 sobre AWS Step Functions

DIO Live Step Functions - 22/12/2021 Serviços AWS utilizados AWS Step Functions AWS Lambda Amazon S3 Amazon Rekognition Amazon DynamoDB Amazon Cloudwa

Cassiano Ricardo de Oliveira Peres 5 Mar 01, 2022
A pixeldrain python package using pixeldrain official api

Made with Python3 (C) @FayasNoushad Copyright permission under MIT License License - https://github.com/FayasNoushad/Pixeldrain/blob/main/LICENSE In

Fayas Noushad 6 Jan 26, 2022