I would like to propose some changes to the directory structure, but these might be totally irrelevant due to my misunderstanding.

Current directory entry is fixed at 17bytes, stores x,y as individual values, and requires x,y sorting order. I think all of those may benefit from a bit of restructuring:

• Combine x and y values into a single interleaved value (z-curve). This will make tiles more locale-clustered, possibly reducing the number of range requests (web).
• Make the combined x,y value size depend on the zoom level, rounding to the nearest byte boundary. zooms 0..3 -- 1 byte, 4..7 -- 2 bytes, etc. All leaf nodes are required to be the same zoom, making the directory entries the same size only inside a single directory block, rather than everywhere.
• For the root dir, it could still be 17bytes (although it seems 6 bytes is a bit too much -- zoom 16..19 only requires 5 bytes, so unless pmtiles wants to support more... and even that can be made flex)

It seems like go-pmtiles is set up to read from cloud storage (S3, Google Storage etc), but the python version here is not. Is that correct? How hard would it be to add that? I use Google, but an S3 example would be great.

If the Reader API is robust, I can add it into a flask/fastapi endpoint pretty trivially, implement my own authentication etc, but it definitely seems like being able to read the tiles from a file in Storage (rather than locally) is the really powerful use case here.

When using pmtiles.js, network performance is a big bottleneck.

This is visible in the leaflet raster demo but mitigated by a very fast backend (read from a small pmtiles file), so response-time is ~15ms per tile. Still you can see in the chrome dev tools that each fetch request is "stalled" (queued) until the previous one is answered.

My setup responds in ~150ms per tile, so time from scrolling to display is several seconds.

How can this be optimised?

• Reducing number of requests using Multipart Ranges does not seem possible without adding complexity to the leaflet integration.
• ... but according to chrome we're supposed to have up to 6 concurrent TCP connections per origin?

Options:

• Gzip compression - requires a library like pako, may be expensive
• Cap'n proto packing: https://capnproto.org/encoding.html
• Protobuf varints: https://developers.google.com/protocol-buffers/docs/encoding

Use cases:

• Dense tile pyramids
• Sparse pyramids (tippecanoe output)

I'm using Cordova/Ionic in order to serve offline tiles. I'm doing this by leveraging sqlite-ext to open a mbtiles file on the device and serve the tiles from there. Is there a way to to use pmtiles in Cordova in similar manor? From the documentation it seems that there's a need to use memory mapped file or something similar, have you experimented with this?

... to avoid extra indirection for as many tiles as we can

Previously, we hardcoded a "base_zoom = 7", which is only only optimal for a whole world map. instead "7" is replaced with min zoom that fits all tiles.

I only tested on one example (Bugianen, 152742 tiles) and got the expected base zoom of 14.

```sh
sqlite3 Bugianen.mbtiles "SELECT  zoom_level, COUNT(*) FROM tiles GROUP BY zoom_level"
12|448
13|1792
14|7168
15|28672
16|114662

PYTHONPATH=$PWD bin/pmtiles-convert Bugianen.mbtiles optim_Bugianen.pmtiles
Num tiles: 152742
Num unique tiles: 151947
Num leaves: 7168

So, this needs more testing.

I'm getting a python AttributeError when converting a .mbtiles file to .pmtiles

pmtiles-convert postcode.mbtiles postcode2.pmtiles
('compression:', 'disabled')
Traceback (most recent call last):
  File "/home/malcolm/.local/bin/pmtiles-convert", line 38, in <module>
    mbtiles_to_pmtiles(args.input, args.output, args.maxzoom, args.gzip)
  File "/home/malcolm/.local/lib/python2.7/site-packages/pmtiles/convert.py", line 40, in mbtiles_to_pmtiles
    writer.write_tile(row[0], row[1], flipped, force_compress(row[3], gzip))
  File "/home/malcolm/.local/lib/python2.7/site-packages/pmtiles/convert.py", line 15, in force_compress
    return gzip.decompress(data)
AttributeError: 'module' object has no attribute 'decompress'

postcode.zip

I'm trying to convert a 66 GB file and it quickly consumes 32GB ram then stops.

Is there a solution to converting large files besides launching a super huge VM to handle conversions?

Hi! I'm new to this project. I tried running some html files in the example section in Safari/Chrome but maps were not shown. I got the following errors: I couldn't figure out why fetch is aborted. Is there something I'm missing here?

Thank you.

Some hosts (like github pages) have maximum file sizes. Alternatives like https://github.com/phiresky/sql.js-httpvfs provide a way to split the tile archive until it is less than that max file size (https://github.com/phiresky/sql.js-httpvfs/blob/master/create_db.sh). Would it be possible for the pmtiles reader and writer to optionally support splitting a pmtiles file?

hi, i'm using our own pmtiles tileset server, the test url is : https://tilesets.urbanease.io/cadastre/64/64102/without_protobuf.pmtiles the location for seeing it is :bayonne, france latlng=[43.492949,-1.474841]

on the viewer, all seems to be ok bit leaflet preview don't work viewer i'm using react and leaflet with the npm protomaps package version 1.19.0

my code is simple const map = useMap(); const url = 'https://tilesets.urbanease.io/cadastre/64/64102/without_protobuf.pmtiles'; const layer = protomaps.leafletLayer({ url: url, id: 'cadastral', }); layer.addTo(map);

canvas are created but seems to be transparent, colors in paint_rules layer are good and opacity ok

i don't understand where is the pb bad pmtiles file? no compatibility with leaflet? thx for help

Hey :)

First of all. Thank you for the great format. It is imho the perfect solution to a problem that will become more and more prevalent in the near future.

I tried to implement my own reader / writer in rust and found myself looking at the first-party implementations quite a lot, because I stumbled upon things that are not really clear in the specification.

I think the project would benefit a lot from having a WAY MORE detailed specification (at least more than something I could print on a single page) and I was wondering whether a contribution on my part would be welcomed.

I took the opportunity and wrote a really rough draft, of the section on the header. Just to give you an impression of what I would envision and get your feedback.

PS: It's also totally fine if you do not think the specification needs to be improved.

This will require a major version bump, as the Source API should return a ReadableStream. This is made possible by Lambda now supporting Node 18.

Should also include #90 API changes as well to use If-Match.

• [ ] fflate readablestream
• [ ] varint readable stream

If the new resource has a new ETag but it shorter than the previous, the server might return 416 Range not Satisfiable, which does not have an accompanying ETag. This will invalidate the header, but present a bunch of errors in the console.

We should unify on using the If-Match header in sending the requests, which will return 412 Precondition Failed in the mismatch case, which is designed for this specific situation.

• [x] SVG tile previews should be zoomable
• [x] Should be able to drill down into leaf directories
• [x] should be able to preview vector tiles in leaflet
• [ ] SVG should be feature-level inspectable
• [ ] map preview should be feature-level inspectable
• [ ] see directory Len and header-level metadata
• [ ] inspect SVG with mismatched extents
• [ ] correctly read tile and map hash states