Media-WebScraper
This program scrapes information and images for movies and TV shows.
Summary
For more information on the program, read the WebScrape_help text file (this can also be accessed while running the program).
For a given list of media, the program will scrape and save general information, images and any episode information for each media.
General Information (default):
Saved as a .txt file
This will scrape general information:
- Title
- Release date
- Runtime
- Genre
- Director
- Cast
- Plot description
Additional information saved:
- Source database used for scrape
- ID for media in source database
- Poster image link
Images (default):
Saved as a .jpg file
This will scrape the poster.
Episode Information (if specified):
Saved as a .csv file
This will scrape information for each episode for a TV show:
- Season number
- Episode number
- Episode title
- Episode air date
- Episode description
Features:
- Multithreaded scraping for media in list to greatly improve the time taken when scraping for large media lists.
- Can generate a media list from folders and files in a specified directory or from user input.
- Can specify save location for scraped data.
- Can specify search tags for media list for a more accurate scrape.
- Can choose to scrape all episode information for a TV show.
- Can detect if data is already scraped which allows for scraping new media from an already scraped list of media very efficient.
- Can recover missing scraped files if one or more are missing without rescraping all data.
- Can retry the scrape before exiting the program if there were any incomplete scrapes (successfully scraped files will not be altered or rescraped).
- Currently only supports scraping data from IMDb.
Usage:
For more information on the program, read the WebScrape_help text file (this can also be accessed while running the program).
Currently a terminal-based program.
Running the program using python:
- Requirements: Python 3.2+ (additional libraries: requests, beautifulsoup4)
Running the program from bundled executable file (created using pyinstaller):
- Requirements: Windows 10
- Creates a 'temp' folder containing extracted libraries and support files in the same location as the program while running.
- The temporary files will delete automatically but if the program is closed abruptly, the files will remain.
- The 'temp' folder can be manually deleted after closing the program.
- (As of pyinstaller v4.7, a one-file bundled executable will leave any temp '_MEIxxxxxx' folders if the program is force closed)
Updates:
For information on version history, read the HISTORY markdown file.
2 Dec 22, 2021
12 Oct 28, 2022
1 Feb 10, 2022
2 Jul 22, 2022
46.4k Jan 3, 2023
16 Oct 8, 2022
6 Aug 10, 2022
18 Dec 14, 2022
2 Jan 15, 2022
11 Nov 19, 2022
73 Dec 03, 2022
5 Dec 01, 2022
1 Dec 26, 2021
1 Jul 04, 2022
3 Mar 07, 2022
0 Jan 07, 2022
4 Oct 25, 2022
1 Oct 23, 2021
793 Jan 05, 2023
4 Jun 02, 2022
3 Feb 27, 2022
50 Jan 05, 2023
95 Jan 07, 2023
1 Dec 07, 2021
150 Oct 15, 2022
3 Feb 18, 2022
10 Aug 18, 2022
2 Oct 31, 2022