Divar.ir Ads Scrapper

Introduction

This project first asynchronously grab Divar.ir Ads and then save to .csv and .xlsx files named data.csv and data.xlsx which contains 'Almost' all features that an Ad has on Divar.ir website.

Usage

python3.9 -m venv venv
source ./venv/bin/activate
pip install -r requirements.txt
python main.py

Parameters

There are some parameters defined at the first of main.py which can modify some behaviours of program.

With higher WORKERS value you get more async workers, but it rises the risk of getting HTTP 429 Too Many Requests response status.

If you want to scrap all pages of the website, put THRESHOLD to an high value.

By default, this script scrap all Ads of apartments in Niavaran and District 1. You can change AD_LIST_URL in order to get another desired Ads.