Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
An Amazon Price Tracker app helps you to buy which product you want within sale price by sending an E-Mail.

Amazon Price Tracker An Amazon Price Tracker app helps you to buy which product you want within sale price by sending an E-Mail. Installing Download t

Aytaç Kaşoğlu 2 Feb 10, 2022
A discord webhook client written in Python.

DiscordWebhook A discord webhook client written in Python. Installation pip install webhook-client Example from webhook_client import WebhookClient, E

Elijah 4 Nov 28, 2022
Linky bot, A open-source discord bot that allows you to add links to ur website, youtube url, etc for the people all around discord to see!

LinkyBot Linky bot, An open-source discord bot that allows you to add links to ur website, youtube url, etc for the people all around discord to see!

AlexyDaCoder 1 Sep 20, 2022
Pixoo-Awesome is a tool to get more out of your Pixoo Devices.

Pixoo-Awesome is a tool to get more out of your Pixoo Devices. It uses the Pixoo-Client to connect to your Pixoo devices and send data to them. I targ

Horo 10 Oct 27, 2022
Docker image for epicseven gvg qq chatbot based on Xunbot

XUN_Langskip XUN 是一个基于 NoneBot 和 酷Q 的功能型QQ机器人,目前提供了音乐点播、音乐推荐、天气查询、RSSHub订阅、使用帮助、识图、识番、搜番、上车、磁力搜索、地震速报、计算、日语词典、翻译、自我检查,权限等级功能,由于是为了完成自己在群里的承诺,一时兴起才做的,所

Xavier Xiong 2 Jun 08, 2022
Dashboard to monitor the performance of your Binance Futures account

futuresboard A python based scraper and dashboard to monitor the performance of your Binance Futures account. Note: A local sqlite3 database config/fu

86 Dec 29, 2022
A Bot Telegram Anti Users Channel to automatic ban users who using channel to send message in group.

Tg_Anti_UsersChannel A Bot Telegram Anti Users Channel to automatic ban users who using channel to send message in group. Features: Automatic ban Whit

idzeroid 6 Dec 26, 2021
Skyscanner Python SDK

Skyscanner Python SDK Important As of May 1st, 2020, the project is deprecated and no longer maintained. The latest update in v1.1.5 includes changing

Skyscanner 118 Sep 23, 2022
An advanced automatic top.gg dank memer voter that votes automatically for you.

Auto Dank Memer Voter An automatic dank memer voter that sends votes onto top.gg every 12 hours, unless their is captcha. I am working on a captcha de

6 Aug 27, 2022
A powerfull Telegram Leech Bot

owner of this repo :- Abijthkutty contact me :- Abijth Telegram Torrent and Direct links Leecher Dont Abuse The Repo ... this is intented to run in Sm

αвιנтн 9 Jun 11, 2022
A file-based quote bot written in Python

Let's Write a Python Quote Bot! This repository will get you started with building a quote bot in Python. It's meant to be used along with the Learnin

1 Feb 03, 2022
GG Dorking is a tool to generate GitHub and Google dorking for pentesters and bug bounty hunters.

GG-Dorking GG Dorking is a python tool to generate GitHub and Google dorking links for pentesters and bug bounty hunters. It will help you to find imp

Eslam Akl 80 Nov 24, 2022
Discord Bot for Genshin Impact Wish Simulating

Genshin Inpact Wish Simulation Discord Bot Bot Links Invite Reddit Official Discord Features Discord embed reaction menu for wishes Simple code scalin

Jeffrey Shum 2 Jan 04, 2023
A WhatsApp Crashing Tool for Termux

CrashW A WhatsApp Crashing Tool For Termux Users Installing : apt update && apt upgrade -y pkg install python3 pkg install git git clone git://gith

Gokul Mahato 20 Dec 27, 2022
AWS Auto Inventory allows you to quickly and easily generate inventory reports of your AWS resources.

Photo by Denny Müller on Unsplash AWS Automated Inventory ( aws-auto-inventory ) Automates creation of detailed inventories from AWS resources. Table

AWS Samples 123 Dec 26, 2022
A Discord Bot - has a few commands. Built using python - Discord.py - RIP.

Discord_Bot A Discord Bot has been built here. It is capable of running a few commands. The below present screenshot should suffice in terms of explai

Manab Kumar Biswas 1 May 22, 2022
ML-Test-Client

ML-Test-Client Introduction What is this? This Test Client App is to be used to crowd-test machine learning models with the goal of finding the best c

11 Jul 15, 2022
Braintree Python library

Braintree Python library The Braintree Python library provides integration access to the Braintree Gateway. TLS 1.2 required The Payment Card Industry

Braintree 230 Dec 18, 2022
PESU Academy Discord Bot built for PESsants and PESts of PES University

PESU Academy Bot PESU Academy Discord Bot built for PESsants and PESts of PES University You can add the bot to your Discord Server using this link. O

Aditeya Baral 0 Nov 16, 2021
Desktop Backup Client for Borg

Vorta Backup Client Vorta is a backup client for macOS and Linux desktops. It integrates the mighty BorgBackup with your desktop environment to protec

BorgBase.com 1.5k Jan 03, 2023