Data is available from https://www.nasdaq.com/market-activity/stocks/gme/historical officially, BUT:

• the filename itself don't look consistent to me with the data available
• date ranges cannot be predetermined (as far as I can see)

See this comment for a working update

In order to actually allow custom components to be added we need to remove the validation performed on the scraper.settings.constants.

Those can be kept for internal reference, for now at least, but it would be better to rethink on how that works, maybe even reorganizing the code.

The validation itself is technically not needed anymore, I think, since every component passes through the manager to be handled, so the names can actually be anything now.

It would also be a good idea to actually define naming and import schemes to follow.

Also, to improve code navigation and import we could change the packages structure to something like

├── components/
│   │   # should import, like it does now, `fetchers` and `writers`,
│   │   # at least for now and include `constants`
│   ├── __init__.py
│   ├── component_base.py
│   ├── base_fetcher.py
│   ├── base_parser.py
│   ├── base_parser.py
│   │
│   │   # one folder for each source to handle
│   ├── <source_name>
│   │   │   # imports classes and constants, especially the source_name
│   │   ├── __init__.py
│   │   ├── <source_name>_fetcher.py
│   │   ├── <source_name>_parser.py
│   │   │   # local constants and a SOURCE_NAME for matching
│   │   └── constants.py
│   │
│   │   # one folder for each output type/writer
│   ├── <output_type>/
│   │   ├── __init__.py
│   │   ├──<writer_name>.py
│   │   │   # local constants and OUTPUT_TYPE for matching
│   │   └──constants.py

This could allow anybody to create a new set of handlers for a source/output type, like for example

├── <my_source>/
│   ├── __init__.py
│   ├── <my_source>_fetcher.py
│   ├── <my_source>_parser.py
│   └── constants.py

# classes creation
from scraper.components import Fetcher  # base class

class MyFetcher(Fetcher):
  pass

from scraper.components import manager
from my_source import MyParser, MyFetcher, SOURCE_NAME

# Add them to the manager so that both the app and the cli can use them
manager.register_handler(SOURCE_NAME, MyFetcher, MyParser)

since the interface will be the same regardless and matching is now done through the manager, working this way it's simpler to set up new set of components and organize them.

we are currently missing a method to actually validate that the settings are there.

on the cli this is currently done through cli.utils.validate_settings but this should be one directly by the Settings object (ideally in App before starting the actual loop) and the errors should be available somewhere to give feedback.

This is also needed to complete #67 properly, providing the option to change the dates

As per title. Needed because yes.

I'm planning on using pytest but we'll see.

to simulate responses for requests we could use responses, that should do the trick.

A bunch of things changed, for contributors, developers wanting to implement this as a library and even for end users. Update the readme before develop->master

Here is an updated (as much as possible) list of available data that can be scraped through the app.

A note about NASDAQ DATA

I've already started working on it, before finding out about the pay wall. They have A BUNCH of data, like all the things but due to the paywall for now ALL THINGS RELATED TO NASDAQ are on halt

Already available

List of already available sources implemented. feel free to open issues on bugs/changes you wanna see implemented

• [x] Historical short volume
  • [x] FINRA
    • direct file access (csv)
    • no authentication needed
    • reported daily
• [x] Historical fail to deliver
  • [x] SEC
    • direct file access (zip file with csv in .txt format)
    • no authentication needed
    • reported twice a month, reports contain daily data

Working on

List of approved suggestions that are work in progress

• [ ] Historical quotes
  • [ ] NASDAQ
    • REST API
    • requires authentication
    • behind pay wall if not requested through browser
  • [ ] YAHOO
    • REST API
    • requires authentication with optional paywall
    • has free tier with 500 requests/month

Suggested

List of noted suggestions

Rejected

List of suggestions rejected with some reason

It may be time to start writing up some proper documentation, at least for the major parts of the package and some example on how to implement new stuff.

implement argparse or something similar to launch directly bypassing the cli. Useful for automated systems and to just relaunch quickly with options

• [x] Implement core command structure with click
• [x] create commands to launch with arguments
• [x] add option to launch the CLI app
• [x] Should allow specifying a custom path for a settings file, to allow for separate configurations for eventual automations

modular components are functioning cli correctly parses the registered modules

the description obviously is not updated. I was thinking about using the preview command of the menu creator which - while it adds the not needed preview box, might just do the trick perfectly.

Working on implementing the preview function to read the description from the module and add custom formatting to allow highlighting and other things, which may be nice.

Remove all the CONSTANT classes and related validation.

Validation will be done using components modules and manager singleton instead, so it should be completely dynamic

• [x] add constant values to each component module ('handlers' and 'writers')
• [x] remove current validation method and replace with manager's generated lists
  • [x] (optional) Add validation functions to manager so it's more compact and friendly to use
• [x] pass test
• [x] implement new handling in cli
• [x] update tests to pass correctly

Finally closes #44 when merged..?

Major changes

• Added base classes for all components and manager Handlers
• Added validation on manager components registration
• Added manager handling for writers
• Refactored some code in app and tests to handle the changes
• Moved components registration in cli app for cleaner package code

Give the ability the option to specify the path to output the data and/or the template when running the run command

• -o | --output for the output path
• -p | --pattern for the filename pattern generator

some things are already set for this to work but the gist is to have the user be able to specify a format using python formatting your {variable} so that it can know in advance what the file name would be.

options for this could/should be:

• start date - with optional formatting
• end date - with optional formatting
• source
• tickers

The template should be set per writer, when possibile, but for now a global template would be ok. The template should be validated when added (i.e. for missing parenthesis or unknown variables requested

Currently there's some kind of validation of settings in place but it's not how it should be ( #70 ). Also there is no proper logging, with levels and everything, to debug what's happening.

The idea as it comes to mind is to have something like the manager (or even inside the manager!) to register some error messages with some additional info to be used either as level and as some kind of traceback.

Errors could have a basic shape of { level: <logging.level>, message: <error-message>} so that they can be outputted with the logging module or better yet loguru which looks awesome and easy to use and wouldn't require much fuss.

The idea is to expand the settings object and give each component/module its own set of settings, while keeping some as global. Reason being that for example if we implement a MYSQL writer there is not need to have a path to output the data but there is need to configure the mysql connection.

Same goes with sources: some require keys ( see #65 ) that can either be stored in a separate file or in the settings.

This would require some refactor of the setting class, maybe creating some base class for basic functionalities and a way to pass around the settings data to each component

This comes in tandem with #67 since I've been requested to be able to launch with arguments, specifying a settings file and the ability to saveload the settings to/from a custom path.

• Give the user the ability to save the settings on a different path
• Give the user the ability to load the settings from a file
  • arguments
  • from cli, typing the path
  • from cli, through a navigator
• Give the user the ability to set a default settings path.

The app itself should already be capable of most of the work since the instance can be created providing a path to use for the settings file, already has a default one (albeit static for now) and both the to_file and from_file methods accept a path to use.

Also we already have proper handling for missing or wrongly formatted settings file.

• [x] refactor CLI to allow saving the settings to another path
• [ ] refactor CLI to change a default settings path
• [x] refactor Settings to be able to have a safe fallback and handle a default path
• [ ] (optional?) store the settings path on a simple file in the project folder so that the app knows where to look for it.