Ralph is a command-line tool to fetch, extract, convert and push your tracking logs from various storage backends to your LRS or any other compatible storage or database backend.

Overview

Ralph

Ralph is a command-line tool to fetch, extract, convert and push your tracking logs (aka learning events) from various storage backends to your LRS or any other compatible storage or database backend.

Quick start guide

Ralph is distributed as a standard python package; it can be installed via pip or any other python package manager (e.g Poetry, Pipenv, etc.):

$ pip install ralph-malph

Once installed, the ralph command should be available in your PATH. Try to invoke the program usage thanks to the --help flag:

$ ralph --help

You should see a list of available commands and global flags for ralph. Note that each command has its own usage that can be invoked via:

$ ralph COMMAND --help

You should substitute COMMAND by the target command, e.g. list, to see its usage.

Documentation

We try our best to maintain an up-to-date reference documentation for this project. If you intend to install, test or contribute to ralph, we invite you to read this documentation and give us feedback if some parts are unclear or your use case is not (or poorly) covered.

Contributing

This project is intended to be community-driven, so please, do not hesitate to get in touch if you have any question related to our implementation or design decisions.

We try to raise our code quality standards and expect contributors to follow the recommandations from our handbook.

License

This work is released under the MIT License (see LICENSE).

Comments
  • ✨(models) add edx navigational event pydantic models

    ✨(models) add edx navigational event pydantic models

    Purpose

    With the validate and select command newly implemented, it is now possible to implement edx tracking logs events pydantic models to validate incoming events in ralph.

    Proposal

    The first events implemented are the easiest (navigation).

    • [x] event models
    • [x] event fixtures
    • [x] event tests
    needs review feature 
    opened by quitterie-lcs 9
  • Add WebSocket to backends documentation

    Add WebSocket to backends documentation

    Purpose

    Add WebSocket to Backends documentation

    Proposal

    WebSocket

    The webSocket backend is read-only and can be used to get real-time events.

    If you use OVH's Logs Data Platform (LDP), you can retrieve a WebSocket URI to test your data stream by following instructions from the official documentation.

    Backend parameters

    • ws_uri: the WebSocket uri (e.g. wss://example.com/websocket)
    improvement WIP 
    opened by p-bizouard 6
  • ✨(converters) add edx server event xapi converter

    ✨(converters) add edx server event xapi converter

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to convert them into xAPI format.

    Proposal

    ServerEventSchema validates the edx server event.

    ServerEventToXapi uses the ServerEventSchema to validate the edx server event and then converts it to xAPI format.

    XapiConverterSelector deserializes the edx events and tries to match a known converter for the event conversion.

    feature 
    opened by SergioSim 6
  • Add helm chart

    Add helm chart

    Purpose

    Add Helm chart for Ralph

    Description

    Created the chart under src/helm I tried to migrate the logic from the .j2 tray files. The version label was replaced as the helm convention for app.kubernetes.io/version The checksum from the secret name was deleted. The annotation checksum/config: was used instead on the secret, the checksum is automatically generated from vault.yaml it help to update the secret object every time a value is updated. Note that the .Chart.AppVersion is better used to track the version deployed and it should reflect the image.tag There is only one vars files values.yaml the vault is under specific file vault.yaml used only for generating the secrets.

    Questions

    • Why do you have two service with the same config? the only difference is the label static
    • What is deployment_stamp ?
    • What is the purpose of prefix ?
    • How would you like to manage env.?

    ToDo

    • [ ] Add README.md

    You can use helm template . under ./src/helm/ralph to check the output of the chart, it will show you the final manifest that will be deployed.

    feature 
    opened by waammar 5
  • Add MongoDB backend

    Add MongoDB backend

    Purpose

    MongoDB is a classical database for LRSs such as LearningLocker or Apereo Learning Analytics Initiative. Supporting this backend seems a must-have for Ralph.

    Proposal

    • [x] add LearningLocker parser
    • [x] add MongoDB backend
    • [x] ~add learning locker parser~ edit: postponed to a new PR
    needs review feature 
    opened by jmaupetit 5
  • ✨(converter) add edx to xapi video converter

    ✨(converter) add edx to xapi video converter

    Purpose

    edX video events can be converted to xAPI for most of them. The interaction events can still not be converted.

    Proposal

    • [x] load_video to initialized
    • [x] play_video to played
    • [x] pause_video to paused
    • [x] stop_video to terminated
    • [x] seek_video to seeked
    improvement needs review FUN 
    opened by quitterie-lcs 4
  • ✨(models) add xAPI page_viewed and page_terminated models

    ✨(models) add xAPI page_viewed and page_terminated models

    Purpose

    These pydantic models can be used for xAPI format validation.

    Proposal

    The page_viewed pydantic model correspond to the edX simple server event. The page_terminated pydantic model correspond to the edX page_close browser event.

    • [X] page_viewed pydantic model
    • [X] page_terminated pydantic model
    • [x] tests
    feature WIP 
    opened by SergioSim 4
  • ✨(models) add edx server event model

    ✨(models) add edx server event model

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to validate them.

    Proposal

    • [x] ServerEventModel
    • [x] ServerEventFactory
    • [x] Tests
    feature WIP 
    opened by SergioSim 4
  • Add test fixtures

    Add test fixtures

    Purpose

    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    feature WIP 
    opened by SergioSim 4
  • Add

    Add "event" test fixture factory

    Purpose

    • Document the different events that are logged as "tracking logs" in the LMS to understand how, were and when things get logged
    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    • [ ] Events documentation
    feature WIP 
    opened by SergioSim 4
  • ✨(cli) add the new

    ✨(cli) add the new "auth" command

    Purpose

    Generating credentials to authenticate to our LRS should be smooth and painless for our users.

    Proposal

    • [x] add a new auth command to ease new user credentials generation
    • [x] add pydantic models for user credentials checking and credentials file validation
    • [x] track authentication issues to the LRS server
    needs review feature 
    opened by jmaupetit 3
  • `make bootstrap` is failing due to missing build dependences

    `make bootstrap` is failing due to missing build dependences

    Bug Report

    Hi all, I'm excited to get started developing with Ralph. Getting set up I ran into an issue, however. On the master branch make bootstrap errors out pip installing psutil due to gcc not being available.

    Expected behavior/code make bootstrap builds the docker images cleanly.

    Actual Behavior An error occurs:

    #0 22.22       creating build/temp.linux-aarch64-cpython-39/psutil
    #0 22.22       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=594 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-cpython-39/psutil/_psutil_common.o
    #0 22.22       C compiler or Python headers are not installed on this system. Try to run:
    #0 22.22       sudo apt-get install gcc python3-dev
    #0 22.22       error: command 'gcc' failed: No such file or directory
    

    Steps to Reproduce

    1. Sync the Ralph master branch
    2. run make bootstrap
    3. The above error occurs

    Environment

    • Ralph version: master branch
    • Platform: Docker Desktop on Mac OS 13.0.1

    Possible Solution I was able to get a successful build by adding the required development tools, see the PR here: https://github.com/bmtcril/ralph/pull/1

    bug 
    opened by bmtcril 1
  • 📝(project) add helm chart tutorial

    📝(project) add helm chart tutorial

    Purpose

    Ralph can be deployed to a Kubernetes cluster using its official Helm Chart. We need a step-by-steph guide to use it.

    Proposal

    This tutorial guides you to the deployment of a complete learning analytics stack to Kubernetes using Helm.

    FUN WIP 
    opened by jmaupetit 1
  • Implement xAPI forum models in `ralph`

    Implement xAPI forum models in `ralph`

    Purpose

    xAPI forum used in ashley models have to be described in ralph for validation purpose and usage of ralph as a library

    Proposal

    • [ ] Define pydantic models associated to each forum learning statement templates
    • [ ] Define selector for each model
    • [ ] Write model and selector tests
    feature 
    opened by quitterie-lcs 0
  • Integrate mypy for Python type checking

    Integrate mypy for Python type checking

    Purpose

    It has been decided that we use typing in all ralph project. We have to integrate a static type checker in our linting toolbox.

    Proposal

    • Integrate mypy in the project (local tooling for development + CI)
    improvement 
    opened by quitterie-lcs 0
  • 🏗️(backends) unify data backends fs

    🏗️(backends) unify data backends fs

    Purpose

    Storage and Database backends have similar interfaces and usage.

    Proposal

    Unify the interfaces of database and storage backends into a single DataBackend interface.

    • [x] add Data and LRS Backend interfaces
    • [x] add FileSystemDataBackend
    needs review feature 
    opened by SergioSim 1
  • Ora2 events

    Ora2 events

    Purpose

    ORA2 events describe the activity for assessments pair reviewing. Edx defined models and their selectors are described with Pydantic.

    Proposal

    Define Pydantic model for:

    • openassessmentblock.create_submission,
    • openassessmentblock.get_peer_submission,
    • openassessmentblock.get_submission_for_staff_grading,
    • openassessmentblock.peer_assess,
    • openassessmentblock.save_submission,
    • openassessmentblock.self_assess,
    • openassessmentblock.staff_assess,
    • openassessmentblock.student_training_assess_example,
    • openassessment.submit_feedback_on_assessments,
    • openassessment.upload_file,
    needs review feature 
    opened by quitterie-lcs 0
Releases(v3.1.0)
  • v3.1.0(Nov 17, 2022)

    Added

    • EdX to xAPI converters for video events

    Changed

    • Improve Ralph's library integration by unpinning dependencies (and prefer ranges)
    • Upgrade fastapi to 0.87.0
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Oct 19, 2022)

    Added

    • Implement edx video browser events pydantic models
    • Create a post endpoint for statements implementing the LRS spec
    • Implement support for the MongoDB database backend
    • Implement support for custom queries when using database backends get method (used in the fetch command)
    • Add dotenv configuration file support and python-dotenv dependency
    • Add host and port options for the runserver cli command
    • Add support for database selection when running the Ralph LRS server
    • Implement support for xAPI statement forwarding
    • Add database backends status checking
    • Add health LRS router
    • Tray: add LRS server support

    Changed

    • Migrate to python-legacy handler for mkdocstrings package
    • Upgrade click to 8.1.3
    • Upgrade elasticsearch to 8.3.3
    • Upgrade fastapi to 0.79.1
    • Upgrade ovh to 1.0.0
    • Upgrade pydantic to 1.9.2
    • Upgrade pymongo to 4.2.0
    • Upgrade python-keystoneclient to 5.0.0
    • Upgrade python-swiftclient to 4.0.1
    • Upgrade requests to 2.28.1
    • Upgrade sentry_sdk to 1.9.5
    • Upgrade uvicorn to 0.18.2
    • Upgrade websockets to 10.3
    • Make backends yield results instead of writing to standard streams (BC)
    • Use pydantic settings management instead of global variables in defaults.py
    • Rename backend and parser parameter environment variables (BC)
    • Make project dependencies management more modular for library usage

    Removed

    • Remove YAML configuration file support and pyyaml dependency (BC)

    Fixed

    • Tray: do not create a cronjobs list when no cronjob has been defined
    • Restore history mixin logger
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Apr 25, 2022)

    Added

    • Implement edx problem interaction events pydantic models
    • Implement edx textbook interaction events pydantic models
    • ws websocket stream backend (compatible with the fetch command)
    • bundle jq, curl and wget in the fundocker/ralph Docker image
    • Tray: enable ralph app deployment command configuration
    • Add a runserver command with basic auth and a whoami route
    • Create a get endpoint for statements implementing the LRS spec

    Changed

    • Upgrade uvicorn to 0.17.4
    • Upgrade elasticsearch to 7.17.0
    • Upgrade sentry_sdk to 1.5.5
    • Upgrade fastapi to 0.73.0
    • Upgrade pyparsing to 3.0.7
    • Upgrade pydantic to 1.9.0
    • Upgrade python-keystoneclient to 4.4.0
    • Upgrade python-swiftclient to 3.13.0
    • Upgrade pyyaml to 6.0
    • Upgrade requests to 2.27.1
    • Upgrade websockets to 10.1
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Jul 15, 2021)

  • v2.0.0(Jul 12, 2021)

    Added

    • xAPI video interacted pydantic models
    • xAPI video terminated pydantic models
    • xAPI video completed pydantic models
    • xAPI video seeked pydantic models
    • xAPI video initialized pydantic models
    • xAPI video paused pydantic models
    • convert command to transform edx events to xAPI format
    • EdX to xAPI converters for page viewed andpage_close events
    • Implement core event format converter
    • xAPI video played pydantic models
    • xAPI page viewed and page terminated pydantic models
    • Implement edx navigational events pydantic models
    • Implement edx enrollment events pydantic models
    • Install security updates in project Docker images
    • Model selector to retrieve associated pydantic model of a given event
    • validate command to lint edx events using pydantic models
    • Support all available bulk operation types for the elasticsearch backend (create, index, update, delete) using the --es-op-type option

    Changed

    • Upgrade elasticsearch to 7.13.2
    • Upgrade python-swiftclient to 3.12.0
    • Upgrade click to 8.0.1
    • Upgrade click-option-group to 0.5.3
    • Upgrade pydantic to 1.8.2
    • Upgrade sentry_sdk to 1.1.0
    • Rename edx models
    • Migrate model tests from factories to hypothesis strategies
    • Tray: switch from openshift to k8s (BC)
    • Tray: remove useless deployment probes

    Fixed

    • Tray: remove version immutable field in DC selector
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Feb 26, 2021)

    Added

    • EdX server event pydantic model and factory
    • EdX page_close browser event pydantic model and factory
    • Tray: allow to specify a self-generated elasticsearch cluster CA certificate

    Fixed

    • Tray: add missing Swift variables in the secret
    • Tray: fix pods anti-affinity selector

    Removed

    • pandas is no longer required
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 4, 2021)

    Added

    • Support for Swift storage backend
    • Use the push command --ignore-errors option to ignore ES bulk import errors
    • The elasticsearch backend now accepts passing all supported client options

    Changed

    • Remove click_log package dependency
    • Upgrade pyyaml to 5.4.1
    • Upgrade pandas to 1.2.1
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 13, 2021)

    Added

    • Implement base CLI commands (list, extract, fetch & push) for supported backends
    • Support for ElasticSearch database backend
    • Support for LDP storage backend
    • Support for FS storage backend
    • Parse (gzipped) tracking logs in GELF format
    • Support for application's configuration file
    • Add optional sentry integration
    • Distribute Arnold's tray to deploy Ralph in a k8s cluster as cronjobs
    Source code(tar.gz)
    Source code(zip)
Owner
France Université Numérique
Collaboratively Building Innovative Digital Learning Services
France Université Numérique
Project scoped command execution to just do your work

Judoka is a command line utility that lets you define project scoped commands and call them through their alias. It lets you just do (= judo) your work.

Eelke van den Bos 2 Dec 17, 2021
Plumbum: Shell Combinators

Plumbum: Shell Combinators Ever wished the compactness of shell scripts be put into a real programming language? Say hello to Plumbum Shell Combinator

Tomer Filiba 2.5k Dec 28, 2022
ghfetch is ai customizable CLI GitHub personal README generator.

ghfetch is ai customizable CLI GitHub personal README generator. Inspired by famous fetch such as screenfetch, neofetch and ufetch, the purpose of this tool is to introduce yourself as if you were a

Alessio Celentano 3 Sep 10, 2021
Command Line Based Todo Script

Todo-CLI Features Full-Fledged Command Line Based Todo List with the following features planned: Interactive Interface OS Notifications Save and Remov

DSC IIEST 5 Nov 17, 2021
CLI helper to install Github releases on your system.

gh-release-install is a CLI helper to install Github releases on your system. It can be used for pretty much anything, to install a formatter in your CI, deploy some binary using an orcherstration to

Jonas L. 28 Nov 06, 2022
👻 Ghoul is an easy to use information service, allowing you to get/add information on someone or something directly from your terminal.

👻 Ghoul is an easy to use information service, allowing you to get/add information on someone or something directly from your terminal. It c

Billy 11 Nov 10, 2021
grungegirl is the hacker's drug encyclopedia. programmed in python for maximum modularity and ease of configuration.

grungegirl. cli-based drug search for girls. welcome. grungegirl is aiming to be the premier drug culture application. it is the hacker's encyclopedia

Eristava 10 Oct 02, 2022
(BionicLambda Universal SHell) A simple shell made in Python. Docs and possible C port incoming.

blush 😳 (BionicLambda Universal SHell) A simple shell made in Python. Docs and possible C port incoming. Note: The Linux executables were made on Ubu

3 Jun 30, 2021
Rich is a Python library for rich text and beautiful formatting in the terminal.

The Rich API makes it easy to add color and style to terminal output. Rich can also render pretty tables, progress bars, markdown, syntax highlighted source code, tracebacks, and more — out of the bo

Will McGugan 41.4k Jan 03, 2023
Notion-cli-list-manager - A simple command-line tool for managing Notion databases

A simple command-line tool for managing Notion List databases. ✨

Giacomo Salici 75 Dec 04, 2022
A CLI tool to disable and enable security standards controls in AWS Security Hub

Security Hub Controls CLI A CLI tool to disable and enable security standards controls in AWS Security Hub. It is designed to work together with AWS S

AWS Samples 4 Nov 14, 2022
A dashboard for your Terminal written in the Python 3 language,

termDash is a handy little program, written in the Python 3 language, and is a small little dashboard for your terminal, designed to be a utility to help people, as well as helping new users get used

Rebecca White 2 Dec 03, 2021
ICMP Reverse Shell written in Python 3 and with Scapy (backdoor/rev shell)

icmpdoor - ICMP Reverse Shell icmpdoor is an ICMP rev shell written in Python3 and scapy. Tested on Ubuntu 20.04, Debian 10 (Kali Linux), and Windows

Jeroen van Kessel 206 Dec 29, 2022
A Python-based command prompt concept which includes windows command emulation.

PythonCMD A Python-based command prompt concept which includes windows command emulation. Current features: echo: Input your message and it will be cl

1 Feb 05, 2022
Simple script to download OTA packages from Realme's endpoint.

Realme OTA Downloader CLI tool (based on this C# program) to create requests to the Realme's endpoint. Requirements Python 3.9. pycryptodome. Installa

Roger Ortiz 64 Dec 28, 2022
A simple CLI productivity tool to quickly display the syntax of a desired piece of code

Iforgor Iforgor is a customisable and easy to use command line tool to manage code samples. It's a good way to quickly get your hand on syntax you don

Solaris 21 Jan 03, 2023
CLI tool that helps manage shell libraries.

shmgr CLI tool that helps manage shell libraries. Badges 📛 project status badges: version badges: tools / frameworks used by test suite (i.e. used by

Bryan Bugyi 0 Dec 15, 2021
A Python package for Misty II development

Misty2py Misty2py is a Python 3 package for Misty II development using Misty's REST API. Read the full documentation here! Installation Poetry To inst

Chris Scarred 1 Mar 07, 2022
Execute shell command lines in parallel on Slurm, S(on) of Grid Engine (SGE), PBS/Torque clusters

qbatch Execute shell command lines in parallel on Slurm, S(on) of Grid Engine (SGE), PBS/Torque clusters qbatch is a tool for executing commands in pa

Jon Pipitone 26 Dec 12, 2022
Text based command line webcam photobooth app

Skunkbooth Why See it in action Usage Installation Run Media location Contributing Install Poetry Clone the repo Activate poetry shell Install dev dep

David Yang 45 Dec 26, 2022