Push Prometheus metrics to VictoriaMetrics or other exporters

Overview

test-all codecov

prometheus-push-client

Push metrics from your periodic long-running jobs to existing Prometheus/VictoriaMetrics monitoring system.

Currently supports pushes directly to VictoriaMetrics:

For pure Prometheus setups, several options are supported:

  • to pushgateway or prom-aggregation-gateway in OpenMetrics format via HTTP. Please read corresponding docs about appropriate use cases and limitations;
  • to StatsD or statsd-exporter in StatsD format via UDP. Prometheus and StatsD metric types are not fully compatible, so currenly all metrics become StatsD gauges, but rate, increase, histogram_quantile and other PromQL functions produce same results as if types never changed.

Install it via pip:

pip install prometheus-push-client

Metrics

This library uses prometheus-client metric implementation, but adds some minor tweaks.

Separate registry

New metric constructors use separate PUSH_REGISTRY as a default, not to interfere with other metrics already defined and monitored in existing projects.

Default labelvalues

With regular prometheus_client, defaults may be defined for either none or all the labels (with labelvalues), but that's not enough. Moreover labelvalues sometimes doesn't work as expected.

We probably want to define some defaults, like hostname, or more importantly, if we use VictoriaMetrics cluster, VictoriaMetrics_AccountID= (try 0 as a default) label must always be set, and metrics without it will be ignored.

Following example shows how to use defaults, and how to override them if necessary.

import prometheus_push_client as ppc


counter1 = ppc.Counter(
    name="c1",
    labelnames=["VictoriaMetrics_AccountID", "host", "event_type"],
    default_labelvalues={
        "VictoriaMetrics_AccountID": 0,
        "host": socket.gethostname(),
    }
)


# regular usage
counter1.labels(event_type="login").inc()

# overriding defaults
counter1.labels(host="non-default", event_type="login").inc()
# same effect as above: defaults are applied in `labelvalues`
# order for "missing" labels in the beginning
counter1.labels("non-default", "login").inc()

Metrics with no labels are initialized at creation time. This can have unpleasant side-effect: if we initialize lots of metrics not used in currently running job, batch clients will have to push their non-changing values in every synchronization session.

To avoid that we'll have to properly isolate each task's metrics, which can be impossible or rather tricky, or we can create metrics with default, non-changing labels (like hostname). Such metrics will be initialized on first use (inc), and we'll be pushing only those we actually utilized.

Clients

Batch clients

Batch clients spawn synchronization jobs "in background" (meaning in a thread or asyncio task) to periodically send all metrics from ppc.PUSH_REGISTRY to the destination.

Clients will attempt to stop gracefully, synchronizing registry "one last time" after job exits or crashes. Sometimes this may mess up sampling, but the worst case I could artifically create looks like this:

graceful push effect

Best way to use them is via decorators / context managers. These clients are intended to be used with long running, but finite tasks, which could be spawned anywhere, therefor not easily accessible by the scraper. If that's not the case -- just use "passive mode" w/ the scraper instead.

def influx_udp_async(host, port, period=15.0):
def influx_udp_thread(host, port, period=15.0):
def statsd_udp_async(host, port, period=15.0):
def statsd_udp_thread(host, port, period=15.0):
def influx_http_async(url, verb="POST", period=15.0):
def influx_http_thread(url, verb="POST", period=15.0):
def openmetrics_http_async(url, verb="POST", period=15.0):
def openmetrics_http_thread(url, verb="POST", period=15.0):

Usage example:

import prometheus_push_client as ppc


req_hist = ppc.Histogram(
    name="external_requests",
    namespace="acme"
    subsystem="job123",
    unit="seconds",
    labelnames=["service"]
)


@ppc.influx_udp_async("victoria.acme.inc.net", 9876, period=15)
async def main(urls):
    # the job ...
    req_hist.labels(gethostname(url)).observe(response.elapsed)

# OR

async def main(urls):
    async with ppc.influx_udp_async("victoria.acme.inc.net", 9876, period=15):
        # the job ...
        req_hist.labels(gethostname(url)).observe(response.elapsed)

Please read about mandatory job tag within url while using pushgateway.

Streaming clients

If for some reason every metric change needs to be synced, UDP streaming clients are implemented in this library.

def influx_udp_aiostream(host, port):
def influx_udp_stream(host, port):
def statsd_udp_aiostream(host, port):
def statsd_udp_stream(host, port):

Usage is completely identical to batch clients' decorators / context managers.

⚠️ Histogram and Summary .time() decorator doesn't work in this mode atm, because it can't be monkey-patched easily.

Transports

Main goal is not to interrupt measured jobs with errors from monitoring code. Therefor all transports will attempt to catch all network errors, logging error info and corresponding tracebacks to stdout.

You might also like...
Google Foobar challenge solutions from my experience and other's on the web.
Google Foobar challenge solutions from my experience and other's on the web.

Google Foobar challenge Google Foobar challenge solutions from my experience and other's on the web. Note: Problems indicated with "Mine" are tested a

Custom component to calculate estimated power consumption of lights and other appliances
Custom component to calculate estimated power consumption of lights and other appliances

Custom component to calculate estimated power consumption of lights and other appliances. Provides easy configuration to get virtual power consumption sensors in Home Assistant for all your devices which don't have a build in power meter.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.
The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss.

The tool helps to find hidden parameters that can be vulnerable or can reveal interesting functionality that other hunters miss. Greater accuracy is achieved thanks to the line-by-line comparison of pages, comparison of response code and reflections.

Emulate and Dissect MSF and *other* attacks
Emulate and Dissect MSF and *other* attacks

Need help in analyzing Windows shellcode or attack coming from Metasploit Framework or Cobalt Strike (or may be also other malicious or obfuscated code)? Do you need to automate tasks with simple scripting? Do you want help to decrypt MSF generated traffic by extracting keys from payloads?

External Network Pentest Automation using Shodan API and other tools.

Chopin External Network Pentest Automation using Shodan API and other tools. Workflow Input a file containing CIDR ranges. Converts CIDR ranges to ind

to learn how to do pull request and do contribution to other's repo
to learn how to do pull request and do contribution to other's repo

Hacktoberfest-2021 - open-source-contribution An Open Source repository to Teach people How to contribute to open sources. 💥 🔥 JOIN PVX PROGRAMMING

python scripts and other files to generate induction encoder PCBs in Kicad
python scripts and other files to generate induction encoder PCBs in Kicad

induction_encoder python scripts and other files to generate induction encoder PCBs in Kicad Targeting the Renesas IPS2200 encoder chips.

scap is a tool for putting code in places and for other purposes

Scap is the deployment script used by Wikimedia Foundation to publish code and configuration on production web servers.

Python script for changing the SSH banner content with other content

Banner-changer-py Python script for changing the SSH banner content with other content. The Script will take the content of a specified file range and

Comments
  • Can I use this to push historical data?

    Can I use this to push historical data?

    I came here from https://stackoverflow.com/a/67562080/7424510.

    Suppose, I've got a temperature sensor, and I've got a list of UNIX time stamps and temperatures, like so:

    [
        (1661863713, 22),
        (1661863714, 21),
        (1661863715, 22)
    ]
    

    Can I push this with this client? How?

    opened by joernschellhaas 1
  • http fixes, pushgateway client

    http fixes, pushgateway client

    • http: ensure data ends with "\n\n"
    • openmetrics: data type metric header
    • added openmetrics http clients for pushgateway / prom-aggregation-gateway
    opened by gistart 1
Releases(0.0.8)
Owner
olegm
olegm
Track testrail productivity in automated reporting to multiple teams

django_web_app_for_testrail testrail is a test case management tool which helps any organization to track all consumption and testing of manual and au

Vignesh 2 Nov 21, 2021
A basic interpreted programming language written in python

shin A basic interpreted programming language written in python. extension You can use our own extension ".shin". Example: main.shin How to start Clon

12 Nov 04, 2022
CRC Reverse Engineering Tool in Python

CRC Beagle CRC Beagle is a tool for reverse engineering CRCs. It is designed for commnication protocols where you often have several messages of the s

Colin O'Flynn 51 Jan 05, 2023
Easytile blender - Simple Blender 2.83 addon for tiling meshes easily

easytile_blender Dead simple, barebones Blender (2.83) addon for placing meshes as tiles. Installation In Blender, go to Edit Preferences Add-ons

Sam Gibson 6 Jul 19, 2022
Vaccine for STOP/DJVU ransomware, prevents encryption

STOP/DJVU Ransomware Vaccine Prevents STOP/DJVU Ransomware from encrypting your files. This tool does not prevent the infection itself. STOP ransomwar

Karsten Hahn 16 May 31, 2022
a simple thing that i made for fun :trollface:

we-do-a-little-trolling about a simple thing that i made for fun. requirements and instructions first you need to install obs , then start the virtual

ranon rat 6 Jul 15, 2022
Script to use SysWhispers2 direct system calls from Cobalt Strike BOFs

SysWhispers2BOF Script to use SysWhispers2 direct system calls from Cobalt Strike BOFs. Introduction This script was initially created to fix specific

FalconForce 101 Dec 20, 2022
A working roblox account generator it doesnt bypass the capcha stuff cuz these didnt showed up in my test runs

A working roblox account generator (state 11.5.2021) it doesnt bypass the capcha stuff cuz these didnt showed up in my test runs

TerrificTable 22 Jan 03, 2023
NBT-Project: This is a APP for building NBT's

NBT-Project This is an APP for building NBT's When using this you select a box on kit maker You input the name and enchant in there related boxes Then

1 Jan 21, 2022
A simple hash system.

PBH-Hash-System A simple hash system. Usage You could use it like this: from pbh import pbh print(pbh("Hey", True)) Output: 2feae2471698cfcdcbd6b98ca

Karim 3 Mar 24, 2022
Course materials for a 3-day seminar "Machine Learning and NLP: Advances and Applications" at New College of Florida

Machine Learning and NLP: Advances and Applications This repository hosts the course materials used for a 3-day seminar "Machine Learning and NLP: Adv

Yoshi Suhara 11 Jun 22, 2022
An audnexus client, providing rich author and audiobook data to Plex via it's legacy plugin agent system.

Audnexus.bundle An audnex.us client, providing rich author and audiobook data to Plex via it's legacy plugin agent system. 📝 Table of Contents About

David Dembeck 248 Jan 02, 2023
一个可以自动生成PTGen,MediaInfo,截图,并且生成发布所需内容的脚本

Differential 差速器 一个可以自动生成PTGen,MediaInfo,截图,并且生成发种所需内容的脚本 为什么叫差速器 差速器是汽车上的一种能使左、右轮胎以不同转速转动的结构。使用同样的动力输入,差速器能够输出不同的转速。就如同这个工具之于PT资源,差速器帮你使用同一份资源,输出不同PT

Lei Shi 96 Dec 15, 2022
A simple wrapper for joy library

Joy CodeGround A simple wrapper for joy library to render joy sketches in browser using vs code, (or in other words, for those who are allergic to Jup

rijfas 9 Sep 08, 2022
Open source stenotype engine

Plover Bringing stenography to everyone. Homepage Releases Wiki Blog Google Group Discord Chat About Installation Getting help Contributing Donations

Open Steno Project 2k Jan 09, 2023
The functions we created are included in a script. The necessary parts for pre-processing were taken. Analysis complete.

Feature-Engineering The functions we created are included in a script. The necessary parts for pre-processing were taken. Analysis complete. Business

Ayşe Nur Türkaslan 4 Oct 17, 2021
A Way to Use Python, Easier.

PyTools A Way to Use Python, Easier. How to Install Just copy this code, then make a new file in your project directory called PyTools.py, then paste

Kamran 2 Aug 15, 2022
Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat

Video Stream is an Advanced Telegram Bot that's allow you to play Video & Music on Telegram Group Video Chat 📊 Stats 🧪 Get SESSION_NAME from below:

dark phoenix 12 May 08, 2022
A simple wrapper to analyse and visualise reinforcement learning agents' behaviour in the environment.

Visrl Visrl (pronounced "visceral") is a simple wrapper to analyse and visualise reinforcement learning agents' behaviour in the environment. Reinforc

Jet New 14 Jun 27, 2022
Mpis-ex7 - Implementation of tasks 1, 2, 3 for Metody Probabilistyczne i Statystyka Lista 7

Implementations of task 1, 2 and 3 from here Author: Maciej Bazela Index: 261743 Each task was implemented in Python 3. I've used Cython to speed up e

Maciej Bazela 1 Feb 27, 2022