A Python package which supports global logfmt formatted logging.

Overview

Python Logfmter

pre-commit test

A Python package which supports global logfmt formatted logging.

Install

$ pip install logfmter

Usage

Before integrating this library, you should be familiar with Python's logging functionality. I recommend reading the Basic Logging Tutorial.

This package exposes a single Logfmter class that can be integrated into the standard library logging system similar to any logging.Formatter.

The provided formatter will logfmt encode all logs. Key value pairs are provided via the extra keyword argument or by passing a dictionary as the log message.

Basic

import logging
from logfmter import Logfmter

handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(Logfmter())

logger = logging.getLogger()
logger.addHandler(handler)

logger.error("hello", extra={"alpha": 1}) # at=ERROR msg=hello alpha=1
logger.error({"token": "Hello, World!"}) # at=ERROR token="Hello, World!"

Customize

You can subclass the formatter to change its behavior.

import logging
from logfmter import Logfmter


class CustomLogfmter(Logfmter):
    """
    Provide a custom logfmt formatter which formats
    booleans as "yes" or "no" strings.
    """

    @classmethod
    def format_value(cls, value):
        if isinstance(value, bool):
            return "yes" if value else "no"

	return super().format_value(value)

handler = logging.StreamHandler(sys.stderr)
handler.setFormatter(CustomLogfmter())

logger = logging.getLogger()
logger.addHandler(handler)

logger.error({"example": True}) # at=ERROR example=yes

Development

Required Software

Refer to the links provided below to install these development dependencies:

Getting Started

Setup

$ <runtimes.txt xargs -n 1 pyenv install -s
$ direnv allow
$ pip install -r requirements/dev.txt
$ pre-commit install
$ pip install -e .

Tests

Run the test suite against the active python environment.

$ pytest

Run the test suite against the active python environment and watch the codebase for any changes.

$ ptw

Run the test suite against all supported python versions.

$ tox

Publishing

Build

$ python -m build

Upload

$ twine upload dist/*
Comments
  • Question: How to include default / reserved attributes in the log message?

    Question: How to include default / reserved attributes in the log message?

    Thank you for making this library available. I have been looking at the tests and docs but could not find out how I can include some of the keys you defined as "RESERVED" in my message, such as asctime. I would like to say::

    >>> logger.info("hello")
    "asctime=2022-04-20 msg='hello'"
    

    without having to pass asctime in as an extra. I have tried the below in my dictconfig but did not work::

            'logfmt': {
                'format': 'on=%(asctime)s level=%(levelname)s module=%(name)s msg=%(message)s',
                'class': 'logfmter.Logfmter',
            },
    

    Is this possible?

    opened by meitham 3
  • Include Native Log Record Attributes

    Include Native Log Record Attributes

    We should be able to configure the logger to always include some default keys. For example, a user may want to always include a time or logger key.

    This functionality is seen in the python-json-logger.

    opened by jteppinette 1
  • Add Python 11 Official Support / Testing

    Add Python 11 Official Support / Testing

    I assume this will work on Python 11 just fine, but let's add the Python 11 image to the test scripts and see if there are any improvements that can be made.

    opened by jteppinette 0
  • Update Structure

    Update Structure

    We should move the logfmter directory inside src as suggested by:

    • https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#choosing-a-test-layout-import-rules
    • https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure%3E

    This is highly recommend and will also support the usage of importlib import mode as recommended here.

    opened by jteppinette 0
  • Exclude Keys

    Exclude Keys

    Some third party libraries may make log calls with keys which we don't want to log. We should be able to exclude these keys from the final log line.

    For example, when overriding uvicorn logging. The color_message key is currently printed (it is used by their custom color formatter). If we could exclude this key, then we could prevent the following output:

    at=INFO msg="Started server process [97819]" color_message="Started server process [%d]"
    at=INFO msg="Waiting for application startup."
    at=INFO msg="Application startup complete."
    at=INFO msg="Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)" color_message="Uvicorn running on %s://%s:%d (Press CTRL+C to quit)"
    
    opened by jteppinette 0
Releases(v0.0.6)
  • v0.0.6(Apr 22, 2022)

  • v0.0.5(Apr 20, 2022)

    • Add support for include native log record attributes in the final log output.
    • Add support for overriding the date format used when formatting the asctime attribute.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.4(Mar 30, 2022)

    • Fix the usage documentation
    • Escape newline characters in all logged values. Previously, you could generate multi-line log statements. This should never be the case.
    • Add support for auto-generating exc_info parameters. If the log record is generated with exc_info, as in when using logging.exception(...), the log message will contain properly formatted exception and traceback information.
    • Add support for logging with an empty message dictionary.
    Source code(tar.gz)
    Source code(zip)
  • v0.0.3(Feb 12, 2022)

  • v0.0.1(Feb 6, 2022)

Summarize LSF job properties by parsing log files.

Summarize LSF job properties by parsing log files of workflows executed by Snakemake.

Kim 4 Jan 09, 2022
pyEventLogger - a simple Python Library for making customized Logs of certain events that occur in a program

pyEventLogger is a simple Python Library for making customized Logs of certain events that occur in a program. The logs can be fully customized and can be printed in colored format or can be stored i

Siddhesh Chavan 2 Nov 03, 2022
This open-source python3 script is a builder to the very popular token logger that is on my github that many people use.

Discord-Logger-Builder This open-source python3 script is a builder to the very popular token logger that is on my github that many people use. This i

Local 4 Nov 17, 2021
Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration!

LDAP Monitor Monitor creation, deletion and changes to LDAP objects live during your pentest or system administration! With this tool you can quickly

Podalirius 500 Dec 28, 2022
Log4j alternative for Python

Log4p Log4p is the most secure logging library ever created in this and all other universes. Usage: import log4p log4p.log('"Wow, this library is sec

Isaak Uchakaev 15 Dec 16, 2022
loghandler allows you to easily log messages to multiple endpoints.

loghandler loghandler allows you to easily log messages to multiple endpoints. Using Install loghandler via pip pip install loghandler In your code im

Mathias V. Nielsen 2 Dec 04, 2021
Track Nano accounts and notify via log file or email

nano-address-notifier Track accounts and notify via log file or email Required python libs

Joohansson (Json) 4 Nov 08, 2021
A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics.

A simple, transparent, open-source key logger, written in Python, for tracking your own key-usage statistics, originally intended for keyboard layout optimization.

Ga68 56 Jan 03, 2023
蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎

蓝鲸日志平台(BK-LOG)是为解决分布式架构下日志收集、查询困难的一款日志产品,基于业界主流的全文检索引擎,通过蓝鲸智云的专属 Agent 进行日志采集,提供多种场景化的采集、查询功能。

腾讯蓝鲸 102 Dec 22, 2022
Rich is a Python library for rich text and beautiful formatting in the terminal.

Rich 中文 readme • lengua española readme • Läs på svenska Rich is a Python library for rich text and beautiful formatting in the terminal. The Rich API

Will McGugan 41.5k Jan 07, 2023
A demo of Prometheus+Grafana for monitoring an ML model served with FastAPI.

ml-monitoring Jeremy Jordan This repository provides an example setup for monitoring an ML system deployed on Kubernetes.

Jeremy Jordan 176 Jan 01, 2023
Stand-alone parser for User Access Logging from Server 2012 and newer systems

KStrike Stand-alone parser for User Access Logging from Server 2012 and newer systems BriMor Labs KStrike This script will parse data from the User Ac

BriMor Labs 69 Nov 01, 2022
Progressbar 2 - A progress bar for Python 2 and Python 3 - "pip install progressbar2"

Text progress bar library for Python. Travis status: Coverage: Install The package can be installed through pip (this is the recommended method): pip

Rick van Hattem 795 Dec 18, 2022
GTK and Python based, system performance and usage monitoring tool

System Monitoring Center GTK3 and Python 3 based, system performance and usage monitoring tool. Features: Detailed system performance and usage usage

Hakan Dündar 649 Jan 03, 2023
The easy way to send notifications

See changelog for recent changes Got an app or service and you want to enable your users to use notifications with their provider of choice? Working o

Or Carmi 2.4k Dec 25, 2022
A simple package that allows you to save inputs & outputs as .log files

wolf_dot_log A simple package that allows you to save inputs & outputs as .log files pip install wolf_dot_log pip3 install wolf_dot_log |Instructions|

Alpwuf 1 Nov 16, 2021
Integrates a UPS monitored by NUT into OctoPrint

OctoPrint UPS This OctoPrint plugin interfaces with a UPS monitored by NUT (Network UPS Tools). Requirements NUT must be configured by the user. This

Shawn Bruce 11 Jul 05, 2022
Soda SQL Data testing, monitoring and profiling for SQL accessible data.

Soda SQL Data testing, monitoring and profiling for SQL accessible data. What does Soda SQL do? Soda SQL allows you to Stop your pipeline when bad dat

Soda Data Monitoring 51 Jan 01, 2023
Json Formatter for the standard python logger

This library is provided to allow standard python logging to output log data as json objects. With JSON we can make our logs more readable by machines and we can stop writing custom parsers for syslo

Zakaria Zajac 1.4k Jan 04, 2023
dash-manufacture-spc-dashboard is a dashboard for monitoring read-time process quality along manufacture production line

In our solution based on plotly, dash and influxdb, the user will firstly generate the specifications for different robots, and then a wide range of interactive visualizations for different machines

Dequn Teng 1 Feb 13, 2022