crypto currency scraping

Related tags

Web Crawlingscrypto
Overview

SCRYPTO

What ?

Crypto currencies scraping

(At the moment, only bitcoin and ethereum crypto currencies are supported)

How ?

A python script is running in a container, and scrape informations (using CoinGecko API) about crypto currencies. Then this informations are send to a SQL database.
(You can also run the script alone with a .env file and a systemd service)

SETUP

  • You can launch this scraper by simply using the "docker-compose.yml" file that i let you on this repo.

Docker-compose file

Here is an example of docker-compose file.

version: "3"
services:
  scrypto:
    image: baldurr/scrypto:latest
    container_name: scrypto
    environment:
      - CRYPTO_LIST=bitcoin,ethereum
      - DEVISE_1=eur
      - DEVISE_2=usd
      - SCRAPE_TIME=300
      - SQL_USER=root2
      - SQL_PASSWORD=mypwd
      - SQL_HOST=192.168.1.20
      - SQL_DB=db_scrypto
      - SQL_PORT=3308
    restart: unless-stopped

Environment variable

List of all currencies available:
"btc", "eth", "ltc", "bch", "bnb", "eos", "xrp", "xlm", "link", "dot", "yfi", "usd", "aed", "ars", "aud", "bdt", "bhd", "bmd", "brl", "cad", "chf", "clp", "cny", "czk", "dkk", "eur", "gbp", "hkd", "huf", "idr", "ils", "inr", "jpy", "krw", "kwd", "lkr", "mmk", "mxn", "myr", "ngn", "nok", "nzd", "php", "pkr", "pln", "rub", "sar", "sek", "sgd", "thb", "try", "twd", "uah", "vef", "vnd", "zar", "xdr", "xag", "xau", "bits", "sats"

Var Usage Info
CRYPTO_LIST List of currencies separated by ',' Max: 2 currencies
DEVISE_1 Name of the 1st currencie defined ex: eur
DEVISE_2 Name of the 1nd currencie defined ex: usd
SCRAPE_TIME Scrape interval in second ex: 300 = 5min
SQL_USER SQL user used
SQL_PASSWORD SQL user password
SQL_HOST SQL host which host the database ex: 192.168.1.20, don't set localhost while this will refer to the scrypto container if you use the docker method
SQL_DB SQL database name This var must be set to 'db_scrypto'
SQL_PORT SQL database port

Insatallation

SQL configuration

To use correctly this image, you must create a database named 'db_scrypto'.

If you use the docker method, connect to the scrypto container:

docker exec -it scrypto /bin/bash

Then connect to the SQL database:

mysql -u myuser -p

Enter your password and display databases like this:

SHOW databases;

If 'db_scrypto' doesn't exist, create it:

CREATE DATABASE db_scrypto;

Then you have to create the tables to store the data.
NOTICE:

  • For the table name, please name it like the crypto currencie name: bitcoin, ethereum
  • For the value columns, set the name of the column like this: value_mycurrencie (ex: value_usd)
CREATE TABLE ethereum (data_id INT NOT NULL AUTO_INCREMENT, time DATETIME, metric VARCHAR(20), value_eur numeric(10,2), value_usd numeric(10,2), PRIMARY KEY(data_id));
CREATE TABLE bitcoin (data_id INT NOT NULL AUTO_INCREMENT, time DATETIME, metric VARCHAR(20), value_eur  numeric(10,2), value_usd numeric(10,2), PRIMARY KEY(data_id));

Now you are ready to collect data

Docker configuration

DockerHub image: https://hub.docker.com/repository/docker/baldurr/scrypto

mkdir scrypto
cd scrypto
wget https://raw.githubusercontent.com/Baldurrr/scrypto/main/docker-compose.yml
docker-compose up -d

Wait a bit and:
docker logs scrypto (will display the api response if the configuration worked)

THE RESULT

In this repo, you will also find a json fill that contain the grafana dashboard configuration

Grafana dashboard example: image

抢京东茅台脚本,定时自动触发,自动预约,自动停止

jd_maotai 抢京东茅台脚本,定时自动触发,自动预约,自动停止 小白信用 99.6,暂时还没抢到过,朋友 80 多抢到了一瓶,所以我感觉是跟信用分没啥关系,完全是看运气的。

Aruelius.L 117 Dec 22, 2022
A pure-python HTML screen-scraping library

Scrapely Scrapely is a library for extracting structured data from HTML pages. Given some example web pages and the data to be extracted, scrapely con

Scrapy project 1.8k Dec 31, 2022
Extract gene TSS site form gencode/ensembl/gencode database GTF file and export bed format file.

GetTss python Package extract gene TSS site form gencode/ensembl/gencode database GTF file and export bed format file. Install $ pip install GetTss Us

laojunjun 6 Nov 21, 2022
Command line program to download documents from web portals.

command line document download made easy Highlights list available documents in json format or download them filter documents using string matching re

16 Dec 26, 2022
A leetcode scraper to compile all questions in leetcode free tier to text file. pdf also available.

A leetcode scraper to compile all questions in leetcode free tier to text file, pdf also available. if new questions get added, run again to get new questions.

3 Dec 07, 2021
Python script who crawl first shodan page and check DBLTEK vulnerability

🐛 MASS DBLTEK EXPLOIT CHECKER USING SHODAN 🕸 Python script who crawl first shodan page and check DBLTEK vulnerability

Divin 4 Jan 09, 2022
:arrow_double_down: Dumb downloader that scrapes the web

You-Get NOTICE: Read this if you are looking for the conventional "Issues" tab. You-Get is a tiny command-line utility to download media contents (vid

Mort Yao 46.4k Jan 03, 2023
Explore scraping with BeautifulSoup!

beautifulsoup-scrape Explore scraping with BeautifulSoup! Part One: Start from Shakespeare As my professor is a poet (yes, and he teaches me data and

Chuqin 2 Oct 05, 2022
a small library for extracting rich content from urls

A small library for extracting rich content from urls. what does it do? micawber supplies a few methods for retrieving rich metadata about a variety o

Charles Leifer 588 Dec 27, 2022
Semplice scraper realizzato in Python tramite la libreria BeautifulSoup

Semplice scraper realizzato in Python tramite la libreria BeautifulSoup

2 Nov 22, 2021
IGLS - Instagram Like Scraper CLI tool

IGLS - Instagram Like Scraper It's a web scraping command line tool based on python and selenium. Description This is a trial tool for learning purpos

Shreshth Goyal 5 Oct 29, 2021
Web Scraping Framework

Grab Framework Documentation Installation $ pip install -U grab See details about installing Grab on different platforms here http://docs.grablib.

2.3k Jan 04, 2023
This Spider/Bot is developed using Python and based on Scrapy Framework to Fetch some items information from Amazon

- Hello, This Project Contains Amazon Web-bot. - I've developed this bot for fething some items information on Amazon. - Scrapy Framework in Python is

Khaled Tofailieh 4 Feb 13, 2022
Newsscraper - A simple Python 3 module to get crypto or news articles and their content from various RSS feeds.

NewsScraper A simple Python 3 module to get crypto or news articles and their content from various RSS feeds. 🔧 Installation Clone the repo locally.

Rokas 3 Jan 02, 2022
A simple Discord scraper for discord bots

A simple Discord scraper for discord bots. That includes sending an guild members ids to an file, Mass inviter for joining servers your bot is in and Fetching all the servers of the bot (w/MemberCoun

3zg 1 Jan 06, 2022
Python script for crawling ResearchGate.net papers✨⭐️📎

ResearchGate Crawler Python script for crawling ResearchGate.net papers About the script This code start crawling process by urls in start.txt and giv

Mohammad Sadegh Salimi 4 Aug 30, 2022
热搜榜-python爬虫+正则re+beautifulsoup+xpath

仓库简介 微博热搜榜, 参数wb 百度热搜榜, 参数bd 360热点榜, 参数360 csdn热榜接口, 下方查看 其他热搜待加入 如何使用? 注册vercel fork到你的仓库, 右上角 点击这里完成部署(一键部署) 请求参数 vercel配置好的地址+api?tit=+参数(仓库简介有参数信息

Harry 3 Jul 08, 2022
A distributed crawler for weibo, building with celery and requests.

A distributed crawler for weibo, building with celery and requests.

SpiderClub 4.8k Jan 03, 2023
A simple python web scraper.

Dissec A simple python web scraper. It gets a website and its contents and parses them with the help of bs4. Installation To install the requirements,

11 May 06, 2022
HappyScrapper - Google news web scrapper with python

HappyScrapper ~ Google news web scrapper INSTALLATION ♦ Clone the repository ♦ O

Jhon Aguiar 0 Nov 07, 2022