This PR adds a client for programmatically accessing the Center for Disease Control's (CDC) Social Vulnerability Index (SVI).

"Social vulnerability refers to the potential negative effects on communities caused by external stresses on human health. Such stresses include natural or human-caused disasters, or disease outbreaks. Reducing social vulnerability can decrease both human suffering and economic loss." [source]

The SVI has been released 5 times (2000, 2010, 2014, 2016, and 2018) and calculates a relative percentile ranking in four themes categories and an overall ranking at a given geographic context and geographic scale. The themes are:

• Socioeconomic
• Household Composition & Disability
• Minority Status & Language
• Housing Type & Transportation

Rankings are calculated relative to a geographic context, state or all states (United States) . Meaning, for example, a ranking calculated for some location at the United States geographic context would be relative to all other locations where rankings was calculated in the United States. Similarly, SVI rankings are calculated at two geographic scales, census tract and county scales. Meaning, the rankings correspond to a county for a census tract. For completeness, for example, if you were to retrieve the 2018 SVI at the census tract scale, at the state context for the state of Alabama, you would receive 1180 records (number of census tracts in AL in 2010 census) where each ranked percentile is calculated relative to census tracts in Alabama. The tool released in this PR only supports querying for ranking calculated at the United States geographic context. Future work will add support for retrieving rankings at the state spatial scale.

Documentation for each year release of the SVI are located below:

• 2000
• 2010
• 2014
• 2016
• 2018

Example

from hydrotools.svi_client import SVIClient

client = SVIClient()
df = client.get(
    location="AL", # state / nation name (i.e. "alabama" or "United States") also accepted. case insensitive
    geographic_scale="census_tract", # "census_tract" or "county"
    year="2018", # 2000, 2010, 2014, 2016, or 2018
    geographic_context="national" # only "national" supported. "state" will be supported in the future
    )
print(df)
                    state_name state_abbreviation  ... svi_edition                                           geometry
        0        alabama                 al  ...        2018  POLYGON ((-87.21230 32.83583, -87.20970 32.835...
        1        alabama                 al  ...        2018  POLYGON ((-86.45640 31.65556, -86.44864 31.655...
        ...          ...                ...  ...         ...                                                ...
        29498    alabama                 al  ...        2018  POLYGON ((-85.99487 31.84424, -85.99381 31.844...
        29499    alabama                 al  ...        2018  POLYGON ((-86.19941 31.80787, -86.19809 31.808...

Additions

• adds a client for programmatically accessing the Center for Disease Control's (CDC) Social Vulnerability Index (SVI)

Testing

1. Integration tests are included that test all valid SVI queries currently supported by the tool.
2. Some unit tests included for utility functions.

Todos

• Future work will add support for retrieving rankings at the state spatial scale.

Checklist

• [x] PR has an informative and human-readable title
• [x] PR is well outlined and documented. See #12 for an example
• [x] Changes are limited to a single goal (no scope creep)
• [x] Code can be automatically merged (no conflicts)
• [x] Code follows project standards (see CONTRIBUTING.md)
• [x] Passes all existing automated tests
• [x] Any change in functionality is tested
• [x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
• [x] Placeholder code is flagged / future todos are captured in comments
• [x] Reviewers requested with the Reviewers tool :arrow_right:

Each time the gcp client is used, it hits gcp to get the requested data. Given the size of the data and that repeated process, it only makes sense to implement some kind of cache. I propose that we use a file db (i.e. sqlite) to accomplish this for simplicity and broad support in python.

High level logic

1. check if db cache has desired data (stored as df)
   1. yes - return data
   2. no - continue
2. get data from gcp
3. create a df from the gcp data
4. cache this df in an sqlite db using the URL path as the key
5. return the df

Requirements

• sqlite lib must support multiprocessing and batch commits

The sqlitedict library is mature, maintained, and seems to fit this bill for this feature. The lib lets you create/connect with a db and use it like you would a python dictionary. Most importantly, it supports multiprocessing.

Hello guys! I have a brand new installation of miniconda3 (version 3.8) on a new system. I am calling the IV module to retrieve NWIS data and see a file in my working folder called:

/[working_folder]/nwisiv_cache.sqlite

How can I tell the program that I want that file in an specific location (i.e. /home)? I do not want this huge file to affect my system, where is this file installed by default?

Thanks! Alex

Issue

hydrotools.nwis_client version 3.0.6 raises a SyntaxError on import. @aaraney would you mind looking into this?

Environment:

Python 3.8.10 (default, Jun  4 2021, 15:09:15)
[GCC 7.5.0] :: Anaconda, Inc. on linux

Duplicate

import hydrotools.nwis_client

Error

'base_url:typing.Union[str, yarl.URL, NoneType]=None' of kind 'POSITIONAL_OR_KEYWORD' follows 'cache:CacheBackend=None' of kind 'KEYWORD_ONLY'
Traceback (most recent call last):
  File "/home/jason.regina/Projects/nwis_test/miniconda3/lib/python3.8/site-packages/aiohttp_client_cache/docs/forge_utils.py", line 32, in wrapper
    return revision(target_function)
  File "/home/jason.regina/Projects/nwis_test/miniconda3/lib/python3.8/site-packages/forge/_revision.py", line 330, in __call__
    next_.validate()
  File "/home/jason.regina/Projects/nwis_test/miniconda3/lib/python3.8/site-packages/forge/_signature.py", line 1344, in validate
    raise SyntaxError(
SyntaxError: 'base_url:typing.Union[str, yarl.URL, NoneType]=None' of kind 'POSITIONAL_OR_KEYWORD' follows 'cache:CacheBackend=None' of kind 'KEYWORD_ONLY'

When retrieving data using the nwis_client tool, something is happening when specifying the startDT and endDT options where the returned data is shift forward in time by 1 hour. May be able to clean-up the date handling and hand-off a lot to pandas.

In relation to #14 and #70. _restclient no longer uses requests_cache.install_cache which monkeypatches the requests library. Instead, requests_cache.CachedSession is now used. enable_cache [bool] parameter added to constructor of RestClient

Changes

• requests_cache.install_cache is no longer used. Instead requests_cache.CachedSession is now used.
• enable_cache [bool] parameter added to constructor of RestClient

Notes

• Passes all unittests, however several slow tests for nwis_client are now failing. There appears to be something happening with multiprocessing and the requests_cache.CachedSession module.

Checklist

• [x] PR has an informative and human-readable title
• [x] PR is well outlined and documented. See #12 for an example
• [x] Changes are limited to a single goal (no scope creep)
• [x] Code can be automatically merged (no conflicts)
• [x] Code follows project standards (see CONTRIBUTING.md)
• [ ] Passes all existing automated tests
• [x] Any change in functionality is tested
• [x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
• [x] Placeholder code is flagged / future todos are captured in comments
• [x] Reviewers requested with the Reviewers tool :arrow_right:

This adds a check and corrects missing boolean categories in Categorical series input to compute_contingency_table.

Additions

• Checks for both True and False in observed and simulated input series.

Removals

• None

Changes

• Update original test to use categorical series instead of raw categories per the original docstring and intended use of compute_contingency_table.

Testing

1. Four new test scenarios for compute_contingency_table that check for all True and all False cases.

Notes

• Fixes #183

Todos

• None

Checklist

• [x] PR has an informative and human-readable title
• [x] PR is well outlined and documented. See #12 for an example
• [x] Changes are limited to a single goal (no scope creep)
• [x] Code can be automatically merged (no conflicts)
• [x] Code follows project standards (see CONTRIBUTING.md)
• [x] Passes all existing automated tests
• [x] Any change in functionality is tested
• [x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
• [ ] Placeholder code is flagged / future todos are captured in comments
• [x] Reviewers requested with the Reviewers tool :arrow_right:

Good morning. I am trying to create a list of events for the past 10 days at station FLRV2

https://water.weather.gov/ahps2/hydrograph.php?wfo=rnk&gage=flrv2

https://waterdata.usgs.gov/nwis/uv?cb_00060=on&cb_00065=on&format=gif_default&site_no=02064000&period=10&begin_date=2021-05-03&end_date=2021-05-10

There has been a gage relocation due to a bridge construction. The function rolling_minimum is failing.

Thanks, Alex

The decision has been made to change the projects name from evaluation_tools to HydroTools. We feel this name better represents/describes the goals of the project and as such, the repo remote name has also been changed prior to this PR.

The bulk of this PR moves both source and test file references and documentation under the new namespace and branding.

Changes

• python package namespace from evaluation_tools to hydrotoolsin source and test files
• branding of package now camel cased as HydroTools
• developer installation procedure changed from pip install -e . to python setup.py develop. The former will no longer work.
• all major package versions bumped:
  • hydrotools: 2.0.0-alpha.0
  • hydrotools.metrics: 1.0.0-alpha.0
  • hydrotools.events: 1.0.0-alpha.0
  • hydrotools.gcp_client: 1.0.0-alpha.0
  • hydrotools.nwis_client: 2.0.0-alpha.0
  • hydrotools._restclient: 2.0.0-alpha.0

Notes

Todos

• fix href to hydrotools._restclient made in 9779ab2 prior to deploy

Checklist

• [x] PR has an informative and human-readable title
• [x] PR is well outlined and documented. See #12 for an example
• [x] Changes are limited to a single goal (no scope creep)
• [x] Code can be automatically merged (no conflicts)
• [x] Code follows project standards (see CONTRIBUTING.md)
• [x] Passes all existing automated tests
• [x] Any change in functionality is tested
• [x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
• [x] Placeholder code is flagged / future todos are captured in comments
• [x] Reviewers requested with the Reviewers tool :arrow_right:

See #180 for bug report. This PR lays the groundwork for two new releases of the hydrotools.nwm_client. The first release, 5.0.1-post1, pins h5netcdf<=0.13.0. This was done to support users who would like to use the software but are dependent on a h5netcdf<=0.13.0. The second release is 5.0.2, this pins h5netcdf>=0.14.0. 5.0.2 also includes a patch that fixes the IndexError propagating from h5netcdf==0.14.0.

fixes #180.

Guidance

• if your application requires h5netcdf<=0.13.0, use hydrotools.nwm_client==5.0.1-post1
• otherwise, use hydrotools.nwm_client>=5.0.2

Changes

• Resolves IndexError: invalid index to scalar variable. present in hydrotools.nwm_client<=5.0.1

Checklist

• [x] PR has an informative and human-readable title
• [x] PR is well outlined and documented. See #12 for an example
• [x] Changes are limited to a single goal (no scope creep)
• [x] Code can be automatically merged (no conflicts)
• [x] Code follows project standards (see CONTRIBUTING.md)
• [ ] Passes all existing automated tests
• [x] Any change in functionality is tested
• [x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
• [x] Placeholder code is flagged / future todos are captured in comments
• [x] Reviewers requested with the Reviewers tool :arrow_right:

Allow users to set an alternative path for the cache when instantiating IVDataService.

Relevant line here: https://github.com/NOAA-OWP/hydrotools/blob/7294aac3cb65001c933b80fcc4a73bd5724f3765/python/nwis_client/src/hydrotools/nwis_client/iv.py#L89

The test runners for #210 found a failing nwm_client_new test. The AttributeError is raised in xarray source when getting a xarray backend. See the below collapsed test failure trace for more detail.

FAILED python/nwm_client_new/tests/test_NWMFileProcessor.py::test_get_dataset - AttributeError: 'EntryPoints' object has no attribute 'get'

=================================== FAILURES ===================================
_______________________________ test_get_dataset _______________________________

Unclosed client session
client_session: <hydrotools._restclient.async_client.ClientSession object at 0x7f90b51bf650>
    def test_get_dataset():
>       ds = NWMFileProcessor.get_dataset(input_directory)

python/nwm_client_new/tests/test_NWMFileProcessor.py:8: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/hydrotools/nwm_client_new/NWMFileProcessor.py:59: in get_dataset
    ds = xr.open_mfdataset(file_list, engine="netcdf4")
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/xarray/backends/api.py:908: in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/xarray/backends/api.py:908: in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/xarray/backends/api.py:[48](https://github.com/NOAA-OWP/hydrotools/actions/runs/3500004127/jobs/5862205193#step:5:49)1: in open_dataset
    backend = plugins.get_backend(engine)
/opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/xarray/backends/plugins.py:161: in get_backend
    engines = list_engines()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    @functools.lru_cache(maxsize=1)
    def list_engines():
>       entrypoints = entry_points().get("xarray.backends", ())
E       AttributeError: 'EntryPoints' object has no attribute 'get'

/opt/hostedtoolcache/Python/3.7.15/x[64](https://github.com/NOAA-OWP/hydrotools/actions/runs/3500004127/jobs/5862205193#step:5:65)/lib/python3.7/site-packages/xarray/backends/plugins.py:105: AttributeError
=============================== warnings summary ===============================
python/nwm_client_new/tests/test_NWMClient.py::test_QueryError
  /opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/hydrotools/nwm_client_new/NWMFileClient.py:197: RuntimeWarning: No data found for configuration 'analysis_assim' and reference time '20T00Z'
    warnings.warn(message, RuntimeWarning)

python/nwm_client_new/tests/test_NWMClient.py::test_QueryError
  /opt/hostedtoolcache/Python/3.7.15/x64/lib/python3.7/site-packages/hydrotools/nwm_client_new/NWMFileClient.py:197: RuntimeWarning: No data found for configuration 'analysis_assim' and reference time '30T01Z'
    warnings.warn(message, RuntimeWarning)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED python/nwm_client_new/tests/test_NWMFileProcessor.py::test_get_dataset - AttributeError: 'EntryPoints' object has no attribute 'get'

A quick look on xarray's issue tracker led me to this -- a verbatim reproduction of this issue. Conversation there pointed to the issue's source, the newly released version (5.0.0) of importlib-metadata and python 3.7 (no longer actively supported by xarray).

pandas >= 1.5.0 was raising FutureWarning in nwis_client.get because of ambiguity in setting a column's values in-place or replacing the column's underlying array. Specifically, the warning is:

FutureWarning: In a future version, 
`df.iloc[:, i] = newvals` will attempt to set the values inplace instead of always setting a new array. To 
retain the old behavior, use either `df[df.columns[i]] = newvals` or, if columns are non-unique, 
`df.isetitem(i, newvals)` dfs.loc[:, "value"] = pd.to_numeric(dfs["value"], downcast="float")

This PR fixes that issue.

fixes #209

nwis_client

Changes

• pandas >= 1.5.0 no longer raised FutureWarning in nwis_client.get. see #209.

Testing

1. test added to verify warning is not raised in nwis_client.get.

Checklist

• [x] PR has an informative and human-readable title
• [x] PR is well outlined and documented. See #12 for an example
• [x] Changes are limited to a single goal (no scope creep)
• [x] Code can be automatically merged (no conflicts)
• [x] Code follows project standards (see CONTRIBUTING.md)
• [ ] Passes all existing automated tests
• [x] Any change in functionality is tested
• [x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
• [x] Placeholder code is flagged / future todos are captured in comments
• [ ] Reviewers requested with the Reviewers tool :arrow_right:

When using the nwis iv client, I receive this warning:

~/env/lib/python3.8/site-packages/hydrotools/nwis_client/iv.py:283: FutureWarning: In a future version, 
`df.iloc[:, i] = newvals` will attempt to set the values inplace instead of always setting a new array. To 
retain the old behavior, use either `df[df.columns[i]] = newvals` or, if columns are non-unique, 
`df.isetitem(i, newvals)` dfs.loc[:, "value"] = pd.to_numeric(dfs["value"], downcast="float")

I received this error when running mypy on a project that depends on hydrotools.nwis_client.

error: Skipping analyzing "hydrotools.nwis_client.iv": module is installed, but missing library stubs or py.typed marker

I discovered that none of the hydryotools subpackages are PEP-561 compliant, however they do use type hints. I think we should considering adding py.typed files to help users that utilize type checkers.

@kvanwerkhoven discovered that the NWM Routelink contains an erroneous mapping from Reach ID 3624261 to USGS Gage 07010040. The error is present in the RouteLink stored on NOMADS and HydroShare. This error affects data returned from the nwm_client and nwm_client_new subpackages. We will update the RouteLink used by HydroTools once it's fixed.

The National Water Model developers have been notified.

Gage: https://waterdata.usgs.gov/mo/nwis/nwismap/?site_no=07010040&agency_cd=USGS

NWM Reach 3624261 is for Deer Creek (contributing area ~6 sq.mi). USGS Gage 07010040 is for Denny Creek (contributing area ~0.5 sq.mi), a tributary to Deer Creek. See the map below (source: water.noaa.gov/map).