Read and write rasters in parallel using Rasterio and Dask

Last update: Aug 30, 2022

Related tags

Overview

dask-rasterio

dask-rasterio provides some methods for reading and writing rasters in parallel using Rasterio and Dask arrays.

Usage

Read a multiband raster

>>> from dask_rasterio import read_raster

>>> array = read_raster('tests/data/RGB.byte.tif')
>>> array
dask.array<stack, shape=(3, 718, 791), dtype=uint8, chunksize=(1, 3, 791)>

>>> array.mean()
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=()>
>>> array.mean().compute()
40.858976977533935

Read a single band from a raster

>>> from dask_rasterio import read_raster

>>> array = read_raster('tests/data/RGB.byte.tif', band=3)
>>> array
dask.array<raster, shape=(718, 791), dtype=uint8, chunksize=(3, 791)>

Write a singleband or multiband raster

>>> from dask_rasterio import read_raster, write_raster

>>> array = read_raster('tests/data/RGB.byte.tif')

>>> new_array = array & (array > 100)
>>> new_array
dask.array<and_, shape=(3, 718, 791), dtype=uint8, chunksize=(1, 3, 791)>

>>> prof = ... # reuse profile from tests/data/RGB.byte.tif...
>>> write_raster('processed_image.tif', new_array, **prof)

Chunk size

Both read_raster and write_raster accept a block_size argument that acts as a multiplier to the block size of rasters. The default value is 1, which means the dask array chunk size will be the same as the block size of the raster file. You will have to adjust this value depending on the specification of your machine (how much memory do you have, and the block size of the raster).

Install

Install with pip:

pip install dask-rasterio

Development

This project is managed by Poetry. If you do not have it installed, please refer to Poetry instructions.

Now, clone the repository and run poetry install. This will create a virtual environment and install all required packages there.

Run poetry run pytest to run all tests.

Run poetry build to build package on dist/.

Issue tracker

Please report any bugs and enhancement ideas using the GitHub issue tracker:

https://github.com/dymaxionlabs/dask-rasterio/issues

Feel free to also ask questions on our Gitter channel, or by email.

Help wanted

Any help in testing, development, documentation and other tasks is highly appreciated and useful to the project.

For more details, see the file CONTRIBUTING.md.

License

Source code is released under a BSD-2 license. Please refer to LICENSE.md for more information.

Download and process satellite imagery in Python using Sentinel Hub services.

Description The sentinelhub Python package allows users to make OGC (WMS and WCS) web requests to download and process satellite images within your Py

659 Dec 23, 2022

An API built to format given addresses using Python and Flask.

An API built to format given addresses using Python and Flask. About The API returns properly formatted data, i.e. removing duplicate fields, distingu

1 Feb 27, 2022

This is a simple python code to get IP address and its location using python

IP address & Location finder @DEV/ED : Pavan Ananth Sharma Dependencies: ip2geotools Note: use pip install ip2geotools to install this in your termin

2 Jul 5, 2022

A package built to support working with spatial data using open source python

EarthPy EarthPy makes it easier to plot and manipulate spatial data in Python. Why EarthPy? Python is a generic programming language designed to suppo

414 Dec 23, 2022

Using SQLAlchemy with spatial databases

GeoAlchemy GIS Support for SQLAlchemy. Introduction GeoAlchemy is an extension of SQLAlchemy. It provides support for Geospatial data types at the ORM

109 Dec 1, 2022

Solving the Traveling Salesman Problem using Self-Organizing Maps

Solving the Traveling Salesman Problem using Self-Organizing Maps This repository contains an implementation of a Self Organizing Map that can be used

3.1k Dec 31, 2022

Example of animated maps in matplotlib + geopandas using entire time series of congressional district maps from UCLA archive. rendered, interactive version below

5 May 18, 2022

Hapi is a Python library for building Conceptual Distributed Model using HBV96 lumped model & Muskingum routing method

Current build status All platforms: Current release info Name Downloads Version Platforms Hapi - Hydrological library for Python Hapi is an open-sourc

15 Dec 26, 2022

Daily social mapping project in November 2021. Maps made using PyGMT whenever possible.

20 Nov 24, 2022

Comments

the time to write a dask array in tif is too long?

I have a image which have 40000cols and 40000 rows ,while I use data = read_raster(filename, band=1), and the use witer_raster(out_filename, data), it need about one minutes, I want to know can it be quickly

opened by DeZhao-Zhang 2
Add a Gitter chat badge to README.md

dymaxionlabs/dask-rasterio now has a Chat Room on Gitter

@munshkr has just created a chat room. You can visit it here: https://gitter.im/dymaxionlabs/dask-rasterio.

This pull-request adds this badge to your README.md:

If my aim is a little off, please let me know.

Happy chatting.

PS: Click here if you would prefer not to receive automatic pull-requests from Gitter in future.

opened by gitter-badger 1
Does dask-rasterio support masked array?
I'm working with dask masked array, and was wondering what would be the translation of these lines?

import rasterio with rasterio.open(inputFile) as source: # this is a 3D numpy array, with dimensions [band, row, col] src_array = source.read(masked=True)

Thank you for your cool lib!
opened by Becheler 0

TypeError: self._hds cannot be converted to a Python object for pickling

Seems that rasterio's _hds object is no more serializable

distributed.protocol.pickle - INFO - Failed to serialize ("('filled-2f9fe0560be0502eda038fa941309294', 0, 0)", <dask_rasterio.write.RasterioDataset object at 0x7f8f9deac828>, (slice(0, 748, None), slice(0, 22415, None)), <unlocked _thread.lock object at 0x7f8f9cb2af58>, False). Exception: self._hds cannot be converted to a Python object for pickling
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/miniconda3/envs/jupyter/lib/python3.6/site-packages/distributed/protocol/pickle.py in dumps(x)
     37     try:
---> 38         result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
     39         if len(result) < 1000:

~/miniconda3/envs/jupyter/lib/python3.6/site-packages/rasterio/_io.cpython-36m-x86_64-linux-gnu.so in rasterio._io.DatasetWriterBase.__reduce_cython__()

TypeError: self._hds cannot be converted to a Python object for pickling

opened by arkanoid87 3

Releases(0.2.1)

0.2.1(Jun 9, 2018)
Fixes a test assertion

Source code(tar.gz)
Source code(zip)
0.2.0(Jun 9, 2018)
Add band keyword argument to read_raster

Source code(tar.gz)
Source code(zip)
0.1.0(Jun 6, 2018)

Source code(tar.gz)
Source code(zip)

Owner

Dymaxion Labs

Creating new value from geospatial imagery with deep learning

GitHub Repository

GetOSM is an OpenStreetMap tile downloader written in Python that is agnostic of GUI frameworks.

GetOSM GetOSM is an OpenStreetMap tile downloader written in Python that is agnostic of GUI frameworks. It is used with tkinter by ProjPicker. Require

3 May 20, 2022

Search and download Copernicus Sentinel satellite images

sentinelsat Sentinelsat makes searching, downloading and retrieving the metadata of Sentinel satellite images from the Copernicus Open Access Hub easy

837 Dec 28, 2022

Program that shows all the details of the given IP address. Build with Python and ipinfo.io API

ip-details This is a program that shows all the details of the given IP address. Build with Python and ipinfo.io API Usage To use this program, run th

4 Mar 01, 2022

Manipulation and analysis of geometric objects

Shapely Manipulation and analysis of geometric objects in the Cartesian plane. Shapely is a BSD-licensed Python package for manipulation and analysis

3.1k Jan 03, 2023

Geographic add-ons for Django REST Framework. Maintained by the OpenWISP Project.

982 Jan 06, 2023

Tools for the extraction of OpenStreetMap street network data

OSMnet Tools for the extraction of OpenStreetMap (OSM) street network data. Intended to be used in tandem with Pandana and UrbanAccess libraries to ex

47 Sep 21, 2022

Fiona reads and writes geographic data files

Fiona Fiona reads and writes geographic data files and thereby helps Python programmers integrate geographic information systems with other computer s

987 Jan 04, 2023

Construct and use map tile grids in different projection.

Morecantile +-------------+-------------+ ymax | | | | x: 0 | x: 1 | | y: 0 | y: 0

67 Dec 23, 2022

Geospatial Image Processing for Python

GIPPY Gippy is a Python library for image processing of geospatial raster data. The core of the library is implemented as a C++ library, libgip, with

83 Aug 19, 2022

A modern, geometric typeface by @chrismsimpson (last commit @ 85fa625 Jun 9, 2020 before deletion)

Metropolis A modern, geometric typeface. Influenced by other popular geometric, minimalist sans-serif typefaces of the new millenium. Designed for opt

183 Dec 25, 2022

GebPy is a Python-based, open source tool for the generation of geological data of minerals, rocks and complete lithological sequences.

GebPy is a Python-based, open source tool for the generation of geological data of minerals, rocks and complete lithological sequences. The data can be generated randomly or with respect to user-defi

16 Nov 29, 2022

prettymaps - A minimal Python library to draw customized maps from OpenStreetMap data.

A small set of Python functions to draw pretty maps from OpenStreetMap data. Based on osmnx, matplotlib and shapely libraries.

9k Jan 08, 2023

Software for Advanced Spatial Econometrics

GeoDaSpace Software for Advanced Spatial Econometrics GeoDaSpace current version 1.0 (32-bit) Development environment: Mac OSX 10.5.x (32-bit) wxPytho

38 Jan 03, 2023

🌐 Local tile server for viewing geospatial raster files with ipyleaflet or folium

🌐 Local Tile Server for Geospatial Rasters Need to visualize a rather large (gigabytes) raster you have locally? This is for you. A Flask application

192 Jan 04, 2023

Python bindings to libpostal for fast international address parsing/normalization

pypostal These are the official Python bindings to https://github.com/openvenues/libpostal, a fast statistical parser/normalizer for street addresses

651 Dec 16, 2022

Starlite-tile38 - Showcase using Tile38 via pyle38 in a Starlite application

Starlite-Tile38 Showcase using Tile38 via pyle38 in a Starlite application. Repo

8 Aug 07, 2022

A ready-to-use curated list of Spectral Indices for Remote Sensing applications.

A ready-to-use curated list of Spectral Indices for Remote Sensing applications. GitHub: https://github.com/davemlz/awesome-ee-spectral-indices Docume

488 Jan 03, 2023

A simple python script that, given a location and a date, uses the Nasa Earth API to show a photo taken by the Landsat 8 satellite. The script must be executed on the command-line.

What does it do? Given a location and a date, it uses the Nasa Earth API to show a photo taken by the Landsat 8 satellite. The script must be executed

42 Nov 26, 2022

Pandas Network Analysis: fast accessibility metrics and shortest paths, using contraction hierarchies :world_map:

Pandana Pandana is a Python library for network analysis that uses contraction hierarchies to calculate super-fast travel accessibility metrics and sh

321 Jan 05, 2023

Automatización de procesos geográficos con Python sobre Esri ArcGIS y QGIS.

Algoritmos y Programación GIS con Python by [email protected] Introducció

7 Dec 28, 2022