crypto currency scraping

Related tags

Web Crawlingscrypto
Overview

SCRYPTO

What ?

Crypto currencies scraping

(At the moment, only bitcoin and ethereum crypto currencies are supported)

How ?

A python script is running in a container, and scrape informations (using CoinGecko API) about crypto currencies. Then this informations are send to a SQL database.
(You can also run the script alone with a .env file and a systemd service)

SETUP

  • You can launch this scraper by simply using the "docker-compose.yml" file that i let you on this repo.

Docker-compose file

Here is an example of docker-compose file.

version: "3"
services:
  scrypto:
    image: baldurr/scrypto:latest
    container_name: scrypto
    environment:
      - CRYPTO_LIST=bitcoin,ethereum
      - DEVISE_1=eur
      - DEVISE_2=usd
      - SCRAPE_TIME=300
      - SQL_USER=root2
      - SQL_PASSWORD=mypwd
      - SQL_HOST=192.168.1.20
      - SQL_DB=db_scrypto
      - SQL_PORT=3308
    restart: unless-stopped

Environment variable

List of all currencies available:
"btc", "eth", "ltc", "bch", "bnb", "eos", "xrp", "xlm", "link", "dot", "yfi", "usd", "aed", "ars", "aud", "bdt", "bhd", "bmd", "brl", "cad", "chf", "clp", "cny", "czk", "dkk", "eur", "gbp", "hkd", "huf", "idr", "ils", "inr", "jpy", "krw", "kwd", "lkr", "mmk", "mxn", "myr", "ngn", "nok", "nzd", "php", "pkr", "pln", "rub", "sar", "sek", "sgd", "thb", "try", "twd", "uah", "vef", "vnd", "zar", "xdr", "xag", "xau", "bits", "sats"

Var Usage Info
CRYPTO_LIST List of currencies separated by ',' Max: 2 currencies
DEVISE_1 Name of the 1st currencie defined ex: eur
DEVISE_2 Name of the 1nd currencie defined ex: usd
SCRAPE_TIME Scrape interval in second ex: 300 = 5min
SQL_USER SQL user used
SQL_PASSWORD SQL user password
SQL_HOST SQL host which host the database ex: 192.168.1.20, don't set localhost while this will refer to the scrypto container if you use the docker method
SQL_DB SQL database name This var must be set to 'db_scrypto'
SQL_PORT SQL database port

Insatallation

SQL configuration

To use correctly this image, you must create a database named 'db_scrypto'.

If you use the docker method, connect to the scrypto container:

docker exec -it scrypto /bin/bash

Then connect to the SQL database:

mysql -u myuser -p

Enter your password and display databases like this:

SHOW databases;

If 'db_scrypto' doesn't exist, create it:

CREATE DATABASE db_scrypto;

Then you have to create the tables to store the data.
NOTICE:

  • For the table name, please name it like the crypto currencie name: bitcoin, ethereum
  • For the value columns, set the name of the column like this: value_mycurrencie (ex: value_usd)
CREATE TABLE ethereum (data_id INT NOT NULL AUTO_INCREMENT, time DATETIME, metric VARCHAR(20), value_eur numeric(10,2), value_usd numeric(10,2), PRIMARY KEY(data_id));
CREATE TABLE bitcoin (data_id INT NOT NULL AUTO_INCREMENT, time DATETIME, metric VARCHAR(20), value_eur  numeric(10,2), value_usd numeric(10,2), PRIMARY KEY(data_id));

Now you are ready to collect data

Docker configuration

DockerHub image: https://hub.docker.com/repository/docker/baldurr/scrypto

mkdir scrypto
cd scrypto
wget https://raw.githubusercontent.com/Baldurrr/scrypto/main/docker-compose.yml
docker-compose up -d

Wait a bit and:
docker logs scrypto (will display the api response if the configuration worked)

THE RESULT

In this repo, you will also find a json fill that contain the grafana dashboard configuration

Grafana dashboard example: image

Goblyn is a Python tool focused to enumeration and capture of website files metadata.

Goblyn Metadata Enumeration What's Goblyn? Goblyn is a tool focused to enumeration and capture of website files metadata. How it works? Goblyn will se

Gustavo 46 Nov 22, 2022
A dead simple crawler to get books information from Douban.

Introduction A dead simple crawler to get books information from Douban. Pre-requesites Python 3 Install dependencies from requirements.txt (Optional)

Yun Wang 1 Jan 10, 2022
Nekopoi scraper using python3

Features Scrap from url Todo [+] Search by genre [+] Search by query [+] Scrap from homepage Example # Hentai Scraper from nekopoi import Hent

MhankBarBar 9 Apr 06, 2022
A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

A Python Covid-19 cases tracker that scrapes data off the web and presents the number of Cases, Recovered Cases, and Deaths that occurred because of the pandemic.

Alex Papadopoulos 1 Nov 13, 2021
Scrapes the Sun Life of Canada Philippines web site for historical prices of their investment funds and then saves them as CSV files.

slocpi-scraper Sun Life of Canada Philippines Inc Investment Funds Scraper Install dependencies pip install -r requirements.txt Usage General format:

Daryl Yu 2 Jan 07, 2022
Simply scrape / download all the media from an fansly account.

Simply scrape / download all the media from an fansly account. Providing updates as long as its continuously gaining popularity, so hit the ⭐ button!

Mika C. 334 Jan 01, 2023
哔哩哔哩爬取器:以个人为中心

Open Bilibili Crawer 哔哩哔哩是一个信息非常丰富的社交平台,我们基于此构造社交网络。在该网络中,节点包括用户(up主),以及视频、专栏等创作产物;关系包括:用户之间,包括关注关系(following/follower),回复关系(评论区),转发关系(对视频or动态转发);用户对创

Boshen Shi 3 Oct 21, 2021
This code will be able to scrape movies from a movie website and also provide download links to newly uploaded movies.

Movies-Scraper You are probably tired of navigating through a movie website to get the right movie you'd want to watch during the weekend. There may e

1 Jan 31, 2022
Automated data scraper for Thailand COVID-19 data

The Researcher COVID data Automated data scraper for Thailand COVID-19 data Accessing the Data 1st Dose Provincial Vaccination Data 2nd Dose Provincia

Porames Vatanaprasan 31 Apr 17, 2022
Quick Project made to help scrape Lexile and Atos(AR) levels from ISBN

Lexile-Atos-Scraper Quick Project made to help scrape Lexile and Atos(AR) levels from ISBN You will need to install the chrome webdriver if you have n

1 Feb 11, 2022
CreamySoup - a helper script for automated SourceMod plugin updates management.

CreamySoup/"Creamy SourceMod Updater" (or just soup for short), a helper script for automated SourceMod plugin updates management.

3 Jan 03, 2022
This repo has the source code for the crawler and data crawled from auto-data.net

This repo contains the source code for crawler and crawled data of cars specifications from autodata. The data has roughly 45k cars

Tô Đức Anh 5 Nov 22, 2022
A simple, configurable and expandable combined shop scraper to minimize the costs of ordering several items

combined-shop-scraper A simple, configurable and expandable combined shop scraper to minimize the costs of ordering several items. Features Define an

2 Dec 13, 2021
A Python Oriented tool to Scrap WhatsApp Group Link using Google Dork it Scraps Whatsapp Group Links From Google Results And Gives Working Links.

WaGpScraper A Python Oriented tool to Scrap WhatsApp Group Link using Google Dork it Scraps Whatsapp Group Links From Google Results And Gives Working

Muhammed Rizad 27 Dec 18, 2022
Crawler do site Fundamentus.com com o uso do framework scrapy, tanto da aba detalhada como a de resumo.

Crawler do site Fundamentus.com com o uso do framework scrapy, tanto da aba detalhada como a de resumo. (Todas as infomações)

Guilherme Silva Uchoa 3 Oct 04, 2022
Twitter Scraper

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely

Tayyab Kharl 45 Dec 30, 2022
Web Scraping Practica With Python

Web-Scraping-Practica Integrants: Guillem Vidal Pallarols. Lídia Bandrés Solé Fitxers: Aquest document és el primer que trobem. A continuació trobem u

2 Nov 08, 2021
让中国用户使用git从github下载的速度提高1000倍!

序言 github上有很多好项目,但是国内用户连github却非常的慢.每次都要用插件或者其他工具来解决. 这次自己做一个小工具,输入github原地址后,就可以自动替换为代理地址,方便大家更快速的下载. 安装 pip install cit 主要功能与用法 主要功能 change 将目标地址转换为

35 Aug 29, 2022
优化版本的京东茅台抢购神器

优化版本的京东茅台抢购神器

1.8k Mar 18, 2022
Python scrapper scrapping torrent website and download new movies Automatically.

torrent-scrapper Python scrapper scrapping torrent website and download new movies Automatically. If you like it Put a ⭐ on this repo 😇 Run this git

Fazil vk 1 Jan 08, 2022