A web application (with multiple API project options) that uses MariaDB HTAP!

Overview

Bookings

Bookings is a web application that, backed by the power of the MariaDB Connectors and the MariaDB X4 Platform, unleashes the power of smart transactions on hundreds of millions of records with lightning fast query performance without having to add any indexes!

This README will walk you through the steps for getting the Bookings web application up and running using MariaDB HTAP. To ensure success, please follow the instructions in order.

Note: The code provided within this repository is completely open source. Please feel free to use it as you see fit.

Table of Contents

  1. Requirements
  2. Introduction to MariaDB
    1. MariaDB Platform
    2. MariaDB SkySQL
  3. Get the code
  4. Set up the HTAP database instance
    1. Get the data, create the schema, and load the data
    2. Set up replication
    3. Create app data
  5. Set up the web application
    1. Anatomy of the app
    2. Build and run the app
  6. Additional resources
    1. Smart transactions
    2. Cross-engine queries
  7. Support and Contribution
  8. License

Requirements

This sample application, no matter which API project you target, will requires the following to be installed/enabled on your machine:

  • MariaDB Client, used to connect to MariaDB instances.
  • Bash (if you are using Windows 10, you will need to enable the Windows Subsystem for Linux), used to run the data download script

Introduction to MariaDB

MariaDB Platform

MariaDB Platform integrates transactional and analytical products so developers can build modern applications by enriching transactions with real-time analytics and historical data, creating insightful experiences and compelling opportunities for customers – and for businesses, endless ways to monetize data.

To get started using MariaDB locally you can:

MariaDB Hybrid Transactional-Analytical Processing (HTAP)

MariaDB Platform supports Hybrid Transactional-Analytical Processing (HTAP) through a combination of MariaDB Enterprise Server, MariaDB ColumnStore, and MariaDB MaxScale.

Here's a simple architecture diagram of MariaDB X4 Platform.

For more details on how to manually deploy MariaDB X4 please refer to the official documentation.

MariaDB SkySQL

SkySQL is the first and only database-as-a-service (DBaaS) to bring the full power of MariaDB Platform to the cloud, including its support for transactional, analytical and hybrid workloads. Built on Kubernetes, and optimized for cloud infrastructure and services, SkySQL combines ease of use and self-service with enterprise reliability and world-class support – everything needed to safely run mission-critical databases in the cloud, and with enterprise governance.

Get started with SkySQL!

Get the code

Download this code directly or use git (through CLI or a client) to retrieve the code using git clone:

$ git clone https://github.com/mariadb-corporation/dev-example-bookings.git

Set up the HTAP database instance

In order to run the Bookings application you will need to have a MariaDB instance to connect to. For more information please check out "Get Started with MariaDB".

Get the data, create the schema, and load the data

This application uses (US domestic) flight data freely available from the Bureau of Transportation on time performance dataset. The [get_flight_data.sh] shell script will be used to download the flight data (between 1990 and 2020) into a folder called data.

Complete the following steps.

  1. Download the flight data (approx. 180 million records, ~30 GB). Depending on your internet connection this may take some time. However, you can simply modify get_flight_data.sh script to adjust the amount of flight information that is downloaded. Doing so will not disrupt subsequent steps.
$ ./get_flight_data.sh
  1. Create the databases and tables load data. Be sure to include your database instance specific information (host url, port number, username, and password)
$ ./create_and_load.sh host_url port user password

Note: Remember to wrap argument values in single quotes if they contain special characters (e.g. !)

By default the create_and_load.sh script has ssl enabled and assumes a MariaDB SkySQL certificate authority chain file exists next to it. Feel free to modify accordingly.

Set up replication

Using MariaDB replication, MariaDB Enterprise Server replicates writes from InnoDB tables to the ColumnStore tables, ensuring that the application can perform analytical processing on current data.

Combining MariaDB replication with MariaDB MaxScale configured as a Binlog Server, MariaDB Enterprise Server can host InnoDB and ColumnStore on the same server.

This application uses replication on a single table called flights, which exists travel.flights (InnoDB) and travel_history.flights (ColumnStore).

Direct access

To set up replication on an HTAP instance you have direct access to add the following replication filter to the MaxScale configuration file (/etc/maxscale.cnf).

[replication-filter]
type         = filter
module       = binlogfilter
match        = /[.]flights/
rewrite_src  = innodb
rewrite_dest = columnstore

For more information on configuring MariaDB HTAP please review the official Enterprise Documentation.

SkySQL

MariaDB SkySQL provides MariaDB Platform for Smart Transactions service, delivering HTAP capabilities. Simply connect to a MariaDB SkySQL (HTAP) instance and execute the following queries.

Create a replication filter.

SELECT set_htap_replication('flights','travel','travel_history');

Confirm a replication filter has been added.

SELECT show_htap_replication();

For more information on configuring HTAP replication for SkySQL please check out the official documentation.

Create app data

This application is merely meant for demonstration purposes so you will need to provide relevant data within the following:

  • travel.flights
  • travel.tickets
  • travel.trips

Creating searchable flights

You will need to supply future flights that can be booked. The process for this is to first add a flight and then create a ticket for that flight. Consider the following example.

An upcoming flight (option) from LAX to ORD on May 5th, 2020.

INSERT INTO `flights` (`year`, `month`, `day`, `day_of_week`, `fl_date`, `carrier`, `tail_num`, `fl_num`, `origin`, `dest`, `crs_dep_time`, `dep_time`, `dep_delay`, `taxi_out`, `wheels_off`, `wheels_on`, `taxi_in`, `crs_arr_time`, `arr_time`, `arr_delay`, `cancelled`, `cancellation_code`, `diverted`, `crs_elapsed_time`, `actual_elapsed_time`, `air_time`, `distance`, `carrier_delay`, `weather_delay`, `nas_delay`, `security_delay`, `late_aircraft_delay`) VALUES (2020, 5, 5, 5, '2020-05-05', 'DL', NULL, 1280, 'LAX', 'ORD', '0600', '0600', NULL, NULL, NULL, NULL, NULL, '0913', '0913', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO `tickets` (`id`, `fl_date`, `fl_num`, `carrier`, `origin`, `dest`, `price`) VALUES (1, '2020-05-05', 1280, 'DL', 'LAX', 'ORD', 240.00);

Creating upcoming trips

Currently you need to create upcoming trips manually. To do this you will need to have a flight, ticket, and trip record. Consider the following example.

An upcoming trip from ORD to LAX on July 4th, 2020.

INSERT INTO `flights` (`year`, `month`, `day`, `day_of_week`, `fl_date`, `carrier`, `tail_num`, `fl_num`, `origin`, `dest`, `crs_dep_time`, `dep_time`, `dep_delay`, `taxi_out`, `wheels_off`, `wheels_on`, `taxi_in`, `crs_arr_time`, `arr_time`, `arr_delay`, `cancelled`, `cancellation_code`, `diverted`, `crs_elapsed_time`, `actual_elapsed_time`, `air_time`, `distance`, `carrier_delay`, `weather_delay`, `nas_delay`, `security_delay`, `late_aircraft_delay`) VALUES (2020, 7, 4, 2, '2020-07-04', 'DL', NULL, 1170, 'ORD', 'LAX', '1420', '1420', NULL, NULL, NULL, NULL, NULL, '1730', '1730', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);

INSERT INTO `tickets` (`id`, `fl_date`, `fl_num`, `carrier`, `origin`, `dest`, `price`) VALUES (2, '2020-07-04', 1170, 'DL', 'ORD', 'LAX', 276.00);

INSERT INTO `trips` (`id`, `ticket_id`) VALUES (1, 2);

Set up the web application

Anatomy of the app

This application is made of two parts:

  • Client
    • communicates with the API.
    • is a React.js project located in the client folder.
  • API
    • uses a MariaDB Connector to connect to MariaDB.
    • contains multiple projects, located in the api folder.

See the README's in client and api for more information on how to get started!

Build and run the app

Additional resources

Smart transactions

At this point you might be wondering, what are smart transactions?

At their core, smart transactions are the standard transactions that databases have been performing for decades – ultimately powering the online interactions we’ve become accustomed to. The difference with modern applications is the use of real-time analytics before, during and/or after these transactions.

Pre-transaction

This application uses real-time analytics before a flight is booked. Each flight ticket option contains information calculated from the historical records (average delay, average duration, flight score, etc.) within the flights table.

Post-transaction

This application also uses real-time analytics after a flight has been booked, and a trip has been created.

Cross-engine queries

This application uses cross-engine queries to maximize the potentials of the MariaDB X4 Platform. Cross-engine querying is the ability to access, via MaxScale, both the transactional and analytics data within a single query.

Support and Contribution

Thanks so much for taking a look at the Bookings app! As this is a very simple example, there's plenty of potential for customization. Please feel free to submit PR's to the project to include your modifications!

If you have any questions, comments, or would like to contribute to this or future projects like this please reach out to us directly at [email protected] or on Twitter.

License

License

Owner
MariaDB Corporation
MariaDB Corporation
Vita Specific Patches and Application for Doki Doki Literature Club (Steam Version) using Ren'Py PSVita

Doki-Doki-Literature-Club-Vita Vita Specific Patches and Application for Doki Doki Literature Club (Steam Version) using Ren'Py PSVita Contains: Modif

Jaylon Gowie 25 Dec 30, 2022
Pytorch implementation of "Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates"

Peer Loss functions This repository is the (Multi-Class & Deep Learning) Pytorch implementation of "Peer Loss Functions: Learning from Noisy Labels wi

Kushal Shingote 1 Feb 08, 2022
Distributed behavioral experiments

Autopilot Docs Paper Forum Hardware Autopilot is a Python framework for performing complex, hardware-intensive behavioral experiments with swarms of n

70 Dec 14, 2022
Refer'd Resume Scanner

Refer'd Resume Scanner I wanted to share a free resource we built to assist applicants with resume building. Our resume scanner identifies potential s

Refer'd 74 Mar 07, 2022
Wordless - the #1 app for helping you cheat at Wordle, which is sure to make you popular at parties

Wordless Wordless is the #1 app for helping you cheat at Wordle, which is sure t

James Kirk 7 Feb 04, 2022
This repo contains scripts that add functionality to xbar.

xbar-custom-plugins This repo contains scripts that add functionality to xbar. Usage You have to add scripts to xbar plugin folder. If you don't find

osman uygar 1 Jan 10, 2022
Visualization of COVID-19 Omicron wave data in Seoul, Osaka, Tokyo, Hong Kong and Shanghai. 首尔、大阪、东京、香港、上海由新冠病毒 Omicron 变异株引起的本轮疫情数据可视化分析。

COVID-19 in East Asian Megacities This repository holds original Python code for processing and visualization COVID-19 data in East Asian megacities a

STONE 10 May 18, 2022
Movie recommend community

README 0. 초록 1) 목적 사용자의 Needs를 기반으로 영화를 추천해주는 커뮤니티 서비스 구현 2) p!ck 서비스란? "pick your taste!" 취향대로 영화 플레이리스트(이하 서비스 내에서의 명칭인 '바스켓'이라 함)를 만들고, 비슷한 취향을 가진

2 Dec 08, 2021
Union oichecklists For Python

OI Checklist Union Auto-Union user's OI Checklists. Just put your checklist's ID in and it works. How to use it? Put all your OI Checklist IDs (that i

FHVirus 4 Mar 30, 2022
Wordle Solver

Wordle Solver Installation Install the following onto your computer: Python 3.10.x Download Page Run pip install -r requirements.txt Instructions To r

John Bucknam 1 Feb 15, 2022
bamboo-engine 是一个通用的流程引擎,他可以解析,执行,调度由用户创建的流程任务,并提供了如暂停,撤销,跳过,强制失败,重试和重入等等灵活的控制能力和并行、子流程等进阶特性,并可通过水平扩展来进一步提升任务的并发处理能力。

bamboo-engine 是一个通用的流程引擎,他可以解析,执行,调度由用户创建的流程任务,并提供了如暂停,撤销,跳过,强制失败,重试和重入等等灵活的控制能力和并行、子流程等进阶特性,并可通过水平扩展来进一步提升任务的并发处理能力。 整体设计 Quick start 1. 安装依赖 2. 项目初始

腾讯蓝鲸 96 Dec 15, 2022
A simple language for new programmers and a toy language ;)

Yell An extremely simple, yet powerful language for new programmers, as well as a toy language ;) Explore the docs » Report Bug · Request Feature Yell

Yell 4 Dec 28, 2021
Small Arrow Vortex clipboard processing library

Description Small Arrow Vortex clipboard processing library. Install You can install this library from PyPI with pip install av-clipboard-lib or compi

Delta Epsilon 1 Dec 18, 2021
a really simple bot that send you memes from reddit to whatsapp

a really simple bot that send you memes from reddit to whatsapp want to use use it? install the dependencies with pip3 install -r requirements.txt the

pai 10 Nov 28, 2021
NORETURN is an esoteric programming language, based around the idea of not going back

NORETURN NORETURN is an esoteric programming language, based around the idea of not going back Concept Program coded in noreturn runs over one array,

1 Dec 15, 2021
Curso de Python 3 do Básico ao Avançado

Curso de Python 3 do Básico ao Avançado Desafio: Buscador de arquivos Criar um programa que faça a pesquisa de arquivos. É fornecido o caminho e um te

Diego Guedes 1 Jan 21, 2022
Used the pyautogui library to automate some processes on the computer

Pyautogui Utilizei a biblioteca pyautogui para automatizar alguns processos no c

Dheovani Xavier 1 Dec 30, 2021
Small projects for python beginners.

Python Mini Projects For Beginners I recently started doing the #100DaysOfCode Challenge in Python. I've used Python before, but I had switched to JS

Sreekesh Iyer 10 Dec 12, 2022
Online-update est un programme python permettant de mettre a jour des dossier et de fichier depuis une adresse web.

Démarrage rapide Online-update est un programme python permettant de mettre a jour des dossier et de fichier depuis une adresse web. Mode préconfiguré

pf4 2 Nov 26, 2021
OB_Template is a vault template reference for using Obsidian.

Obsidian Template OB_Template is a vault template reference for using Obsidian. If you've tested out Obsidian. and worked through the "Obsidian Help"

323 Dec 27, 2022