Custom SLURM wrapper scripts to make finding job histories and system resource usage more easily accessible

Overview

SLURM Wrappers

Executables

job-history

A simple wrapper for grabbing data for completed and running jobs.

nodes-busy

Developed for the HPC systems at University of Arizona. High memory nodes are differentiated from standard nodes with AvailableFeatures=hi_mem.

nodes-busy.mp4
You might also like...
Projeto job insights - Projeto avaliativo da Trybe do Bloco 32: Introdução à Python

Termos e acordos Ao iniciar este projeto, você concorda com as diretrizes do Código de Ética e Conduta e do Manual da Pessoa Estudante da Trybe. Boas

NUM Alert - A work focus aid created for the Hack the Job hackathon

Contributors: Uladzislau Kaparykha, Amanda Hahn, Nicholas Waller Hackathon Team Name: N.U.M General Purpose: The general purpose of this program is to

Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

This is a Poetry plugin that will make it possible to build projects using custom TOML files

Poetry Multiproject Plugin This is a Poetry plugin that will make it possible to build projects using custom TOML files. This is especially useful whe

Coinloggr - A learning resource and social platform for the coin collecting community
Coinloggr - A learning resource and social platform for the coin collecting community

Coinloggr A learning resource and social platform for the coin collecting commun

python scripts - mostly automation scripts

python python scripts - mostly automation scripts You can set your environment in various ways bash #!/bin/bash python - locally on remote host #!/bi

A simple bot that will help you in your learning and make it more fun.

hyperskill-SimpleChattyBot-python A simple bot that will help you in your learning and make it more fun. Syntax bot.py Stages Stage #1: Zuhura Bot we

Make after-work Mending More flexible In Python

Mending Make after-work Mending More flexible In Python A Lite Package focuses on making project's after-post mending pythonic and flexible. Certainly

Comments
  • KeyError: GPUAlloc error when running nodes-busy

    KeyError: GPUAlloc error when running nodes-busy

    Howdy,

    Blake J pointed us (UAB Research Computing) to your nice slurm-wrapper repo :-)

    While job-history and system-busy work great on our cluster, nodes-busy is crashing on our cluster with the following:

    ❯ ./bin/nodes-busy
    
    nodes-busy: visualize live system resource usage.
    Trouble seeing the output? Try 'nodes-busy --ascii'
    
    Traceback (most recent call last):
      File "./bin/nodes-busy", line 766, in <module>
        merged = merge(job_data, node_data)
      File "./bin/nodes-busy", line 330, in merge
        nodes_dictionary[node]["JOBS"][job] = {"CPUs":cpus, "GPUs":jobs_dictionary[job]["GPUAlloc"],"EndTime":jobs_dictionary[job]["EndTime"],"Partition":jobs_dictionary[job]["Partition"],"Restarts":jobs_dictionary[job]["Restarts"]}
    KeyError: 'GPUAlloc'
    

    I'm digging through the code, but figured I'd post this for tracking purposes.

    Thanks, Mike

    opened by flakrat 7
  • nodes-busy

    nodes-busy "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 53196: invalid start byte"

    Howdy, noticed the following error today when running nodes-busy on our cluster.

    > nodes-busy
    
    Traceback (most recent call last):
      File "/share/apps/rc/bin/nodes-busy", line 1177, in <module>
        job_data = get_scontrol_job_data()
      File "/share/apps/rc/bin/nodes-busy", line 370, in get_scontrol_job_data
        output = out.decode('utf-8').split("\n")
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 53196: invalid start byte
    

    I have no idea what's in position 53196, but I fixed the issue with the following addition to output.decode to ignore errors:

    diff --git a/bin/nodes-busy b/bin/nodes-busy
    index 6fb1744..aee728f 100755
    --- a/bin/nodes-busy
    +++ b/bin/nodes-busy
    @@ -367,7 +367,7 @@ def get_scontrol_job_data(target_job = None):
             sys.exit(1)
    
         # Split up space-delimited output into a job dictionary
    -    output = out.decode('utf-8').split("\n")
    +    output = out.decode('utf-8', 'ignore').split("\n")
         for job in output:
             details = job.split(' ')
             for i in details:
    
    opened by flakrat 3
  • Feature added user arg to past-jobs

    Feature added user arg to past-jobs

    Added new argument to past-jobs to support selecting jobs by someone other than the user running the script:

    $ past-jobs -d 30 -u flakrat
    
                             Jobs submitted by user flakrat in last 30 days.
    
    JobID    Start       User            JobName         Partition  Account    State       ExitCode
    ------------------------------------------------------------------------------------------------
    14364561 2022-06-02  flakrat         hostname        express    flakrat    CANCELLED        0:0
    14385956 2022-06-05  flakrat         hostname        express    flakrat    COMPLETED        0:0
    14850816 2022-06-12  flakrat         hostname        express    flakrat    COMPLETED        0:0
    
    opened by flakrat 0
Releases(v0.2.0)
  • v0.2.0(May 11, 2022)

  • v0.1.0(Dec 9, 2021)

    The scripts in this repository have some bits and pieces that are specific to the HPC setup at University of Arizona. These parts are being cleaned up.

    Source code(tar.gz)
    Source code(zip)
Owner
Sara
HPC Consultant at University of Arizona, UITS
Sara
These are After Effects and Python files that were made in the process of creating the video for the contest.

spirograph These are After Effects and Python files that were made in the process of creating the video for the contest. In the python file you can qu

91 Dec 07, 2022
Streamlit — The fastest way to build data apps in Python

Welcome to Streamlit 👋 The fastest way to build and share data apps. Streamlit lets you turn data scripts into sharable web apps in minutes, not week

Streamlit 22k Jan 06, 2023
Powering up Apache JMeter with Streamlit and opening the door for machine learning.

Powering up Apache JMeter with Streamlit Overview Apache JMeter is an open source load testing tool written in 100% pure Java. JMeter supports umpteen

NaveenKumar Namachivayam ⚡ 16 Aug 24, 2022
Application launcher and environment management

Application launcher and environment management for 21st century games and digital post-production, built with bleeding-rez and Qt.py News Date Releas

10 Nov 03, 2022
En este repositorio pondré archivos graciositos de python que hago de vez en cuando

🐍 Apuntes de python 🐍 ¿Quién soy? 👽 Saludos,mi nombre es Carlos Lara. Pero mi nickname en internet es Hercules Kan. Soy un programador autodidacta

Carlos E. Lara 3 Nov 16, 2021
Beancount Importers for DKB (Deutsche Kredit Bank) CSV Exports

Beancount DKB Importer beancount-dkb provides an Importer for converting CSV exports of DKB (Deutsche Kreditbank) account summaries to the Beancount f

Siddhant Goel 24 Aug 06, 2022
Fonts used to be an install-and-forget thing, but many of are now updated regularly.

Your font manager. Fonts used to be an install-and-forget thing, but many of are now updated regularly. fontman helps you keep track of the fonts you

Nico Schlömer 20 Feb 07, 2022
This is a simple quizz which can ask user for login/register session, then consult to the Quiz interface.

SIMPLE-QUIZ- This is a simple quizz which can ask user for login/register session, then consult to the Quiz interface. By CHAKFI Ahmed MASTER SYSTEMES

CHAKFI Ahmed 1 Jan 10, 2022
Just RESTing

petnica-api-workshop Just RESTing Setup Using pipenv You can setup this project with pipenv if you want isolated libraries. After you've installed pip

Aleksa Tešić 1 Oct 23, 2021
Hotpile: High Order Turing Machine Language Compiler

Hotpile: High Order Turing Machine Language Compiler Build and Run Requirements: Python 3.6+, bison, flex, and GCC installed. Needs to be run under UN

Jiang Weihao 4 Dec 29, 2021
Python 3 script for installing kali tools on your linux machine

Python 3 script for installing kali tools on your linux machine

gh0st 2 Apr 20, 2022
BOHB tune library template (included example)

BOHB-template 실행 방법 python main.py 2021-10-10 기준 tf keras 버전 (tunecallback 방식) 완료 tf gradienttape 버전 (train_iteration 방식) 완료 pytorch 버전은 구현 준비중 방법 소개

Seungwoo Han 5 Mar 24, 2022
A simple code for processing images to local binary pattern.

This figure is gotten from this link https://link.springer.com/chapter/10.1007/978-3-030-01449-0_24 LBP-Local-Binary-Pattern A simple code for process

Happy N. Monday 3 Feb 15, 2022
My programming language named JoLang. (Mainly created for fun)

JoLang status: not ready So this is my programming language which I decided to name 'JoLang' (inspired by Jonathan and GoLang). Features I implemented

Jonathan 14 Dec 22, 2022
Solcast Integration for Home Assistant

Solcast Solar Home Assistant(https://www.home-assistant.io/) Component This custom component integrates the Solcast API into Home Assistant. Modified

Greg 45 Dec 20, 2022
Mannaggia is a python application to praise or more likely to curse the saints

Mannaggia-py 👼 Remember Mannaggia? This is a Python remake of it, with new features. mannaggia is a python application to praise or more likely to cu

Christian Visintin 9 Aug 12, 2022
Hitchhikers-guide - The Hitchhiker's Guide to Data Science for Social Good

Welcome to the Hitchhiker's Guide to Data Science for Social Good. What is the Data Science for Social Good Fellowship? The Data Science for Social Go

Data Science for Social Good 907 Jan 01, 2023
Clackety Keyboards Powered by Python

KMK: Clackety Keyboards Powered by Python KMK is a feature-rich and beginner-friendly firmware for computer keyboards written and configured in Circui

KMK Firmware 780 Jan 03, 2023
This is where I learn machine learning

This is where I learn machine learning🤷‍ This means that this repo covers no specific topic of machine learning or a project - I work in here when I want to learn/try something

Wilhelm Berghammer 47 Nov 16, 2022
A complete python calculator with 2 modes Float and Int numbers.

Python Calculator This program is made for learning purpose. Getting started This Program runs using python, install it via terminal or from thier ofi

Felix Sanchez 1 Jan 18, 2022