zylepe

Vibes are all that matter. As soon as you start measuring it, that measurement becomes a target and vendors start optimizing for it at expense of the general usefulness of the model. We’ve seen plenty of models with great benchmark scores flop when people start using it.

Cool! I definitely felt the pain of current options when I added parquet support to Planetiler to process overture data. I ended up using parquet-floor to trim the dependencies but it’s a bit of a hacky approach. If there’s a way to use the lower level utilities from my own threads without hardwood spawning it’s own then I’ll have to give it a shot.

Planetiler currently supports generating MLT by adding —-tile-format=mlt cli argument. It’s only on latest main right now but I should be able to get a release out in the next few days. In my testing I’ve seen ~10% reduction in overall OpenMapTiles archive size with default settings but there are some more optimizations the team is working on that should bring it down even further.

Pole of Inaccessibility is also a useful technique for placing the label for a polygon at a visually pleasing location on a map. @mourner came up with a more efficient algorithm for computing the point https://blog.mapbox.com/a-new-algorithm-for-finding-a-visual... (https://github.com/mapbox/polylabel) which JTS's MaximumInscribedCircle utility is based on, which I use for "innermost point" label placement in planetiler.

I haven’t used markdown in javadoc yet but this seems like at least 3/10? I often want to put paragraphs or bulleted lists in javadoc and find myself wanting to use markdown syntax for readability in the code but need to switch to less readable html tags for tooling to render it properly.

A friend of mine is on a 25 year running every day streak. He flew to Australia and landed 2 days after taking off and said that day “never existed for him” ¯\_(ツ)_/¯

I’ve stopped at that Lee McDonald’s and remember thinking the price seemed kind of high. Little did I know it was the most expensive in the country. Even the eastbound one across I-90 is $1.40 cheaper!

I’m looking forward to be able to memory-map an entire large file without having to split it up into 2gb segments, and to be able to reliably unmap it when done. So many hacks to work around this lack of functionality today…

I came up with an approach for placing labels where each gets a min zoom to show up at such that there are no collisions and always appears beyond that. Personally I like the feel of it a lot better, but the downside is that at a given zoom labels are less dense than if they could show/hide to fill in empty spaces. Also, it makes it impossible to dynamically style the labels, or rotate/tilt.

Demo: https://msbarry.github.io/maplibre-minzoom-demo/world

The protomaps basemap is also built on planetiler: https://github.com/protomaps/basemaps - and Brandon is one of the main contributors a to planetiler!

Exactly, this use case is write-once then read-only after that. Random updates to the sqlite file after that would need more “normal” settings, but they wouldn’t also need as much throughput.

I spent a while optimizing sqlite inserts for planetiler, this is what I came up with:

https://github.com/onthegomap/planetiler/blob/db0ab02263baaa...

It batches inserts into bulk statements and is able to do writes in the 500k+ per second range, and reads are 300-400k/s using those settings.

I built planetiler (https://github.com/onthegomap/planetiler) for this purpose. The output up to z14 is ~80gb and depending on how big of a machine you have it takes from 30 minutes up to a few hours - no DB required, just java or docker. If you are only going to z11-12, it should be quite a bit faster/smaller.

Brandon from Protomaps is also helping add pmtiles output natively to planetiler, so you won't need a conversion step afterwards!

You may have chosen Valhalla for other reasons, but FWIW I’ve found Graphhopper can handle 1-2 orders of magnitude more qps than Valhalla on the same hardware.

We’ve been experimenting with R2 for serving tiles from an 80gb pmtiles archive, unfortunately the latency on up to 3 byte range requests from a worker is 5-10 seconds so it’s not practical for real usage yet. Can we expect that to improve as R2 gets closer to general availability?

You can track progress for planetiler producing pmtiles output here: https://github.com/onthegomap/planetiler/issues/98.

Another bonus of writing pmtiles directly is that it will cut the time to generate vector tiles for the planet from ~40 to ~30 minutes on a 64-core ec2 instance since pmtiles format is much simpler and more efficient to write than SQLite. It should also cut the output file size a bit from the 80gb that mbtiles uses.

Another option if you want to offload the tiles from embedded in your docker image to static file hosting would be to convert the mbtiles output to pmtiles [1]. Then just upload the pmtiles file to s3 or similar and use the maplibre plug-in to read tiles directly from that large file using byte range requests without a tile server.

There’s a python utility to convert mbtiles to pmtiles, but I’m also planning to add native pmtiles output to planetiler soon [2].

[1] https://github.com/protomaps/PMTiles

[2] https://github.com/onthegomap/planetiler/issues/98

Nice! I was going to point out the project I’ve been working on [1] to reduce the cost but I see you’ve already found it :-) Feel free to reach out it you run into any issues!

[1] https://github.com/onthegomap/planetiler

I recently found this approach: https://github.com/onthegomap/planetiler/blob/main/planetile... but I’m hoping this api will eventually let you do that without an external dependency or writing/compiling C connectors.

I have a google sheet where I keep track of each fund I’ve bought and sold. From there I added a column with gain/loss for each tax lot with remaining shares, and a script that runs once a day to email me if there’s an opportunity for tax loss harvesting. It works pretty well.

I like the idea of rebalance opportunity notifications, but it’s harder to get retirement funds into a google sheet with all the automatic contributions that happen…

I make a photo book once a year with the best photos from the year. I use Amazon photos, google photos, etc but the hard copy book is the only one I’d expect to still be around in 100 years.

For an offsite backup, I should start printing 2 and storing one copy at a relative’s house…

Yep, that first one will do it! And you probably want to rip out the polygon merging code and minimum zoom on the attrs. You can use --only-layers= or --exclude-layers= if you don't want all the others.

The main driver of memory usage (~1.0x the input file size) is storing node locations so we can convert the list of node IDs on each way into lat/lon coordinates. By default nodes are stored in a memory-mapped file so it can run with less than 1.5x the input file size, just slower because of all the page faults. Try running with -Xmx=16g to leave 16GB free for memory-mapped file. it will print % complete as it works through the ways in pass2 so you can decide whether it's worth it to keep waiting.

Also you could try switching from the default --nodemap-type=sortedtable to --nodemap-type=sparsearray. Sorted table uses 12 bytes per node and is more compact for extracts, sparse array uses 8 bytes per node but wastes some storage when there are gaps in the ID space.

There is also an osm.pbf extension that embeds node locations in ways, but you need just as much RAM to convert a file to that format: https://docs.osmcode.org/osmium/latest/osmium-add-locations-...

Once you generate the mbtiles file, you can serve from a much smaller machine.

Nothing is foolproof, but for web-based traffic, proper CORS configuration or rejecting traffic based on the referrer (or lack thereof) would go a long way.

Flatmap calls into user-defined profiles in 2 places: first when processing each input element to map it to a vector tile feature, then a second time right before emitting all vector features in a layer.

That second call lets you manipulate vector features on each tile using JTS geometry utilities (i.e. merge nearby polygons or lines with the same tags). PostGIS uses GEOS which is ported from JTS so you have access to pretty much the same geometry utilities - often with the same name.

It's definitely not as flexible as a PostGIS based solution - especially if you join faraway features that don't appear on the same tile, but in practice it was enough to port the entire OpenMapTiles schema.

See the basemap layers package: https://github.com/onthegomap/flatmap/tree/main/flatmap-base... for example see the landcover layer https://github.com/onthegomap/flatmap/blob/main/flatmap-base... which was ported from the SQL contained in https://github.com/openmaptiles/openmaptiles/tree/master/lay...

Also, this OpenMapTiles PR might help improve your existing pipeline after it gets merged: https://github.com/openmaptiles/openmaptiles-tools/pull/383

Edit: formatting

I generated the planet a few times in testing:

with z13 building merge (extra 14 CPU hours of processing):

  - $4.68 on Digital Ocean 128GB 16 CPU instance (4 hours @ $1.17/hr)

  - $2.75 on AWS EC2 c5ad.16xlarge instance (1 hour @ $2.75/hr) 
  - $2.88 on Linode 128GB 50 dedicated CPU instance (2 hours @ $1.44/hr)

Edit: formatting

Thanks! The default "basemap" style is ported from OpenMapTiles v3.12.2.

Thanks! Glad you find it useful!

Thanks! Tilemaker is a great tool and probably the closest alternative (no intermediate db). The most recent benchmarks I saw though were around a day for the planet, so roughly 10x longer than flatmap. If

Edit: benchmark link https://github.com/systemed/tilemaker/issues/315#issue-99432...

Creator here. I maintain https://onthegomap.com as a side project. I originally used Google Maps API but after their price hike switched to OpenStreetMap data. I self-host GraphHopper for routing and tried to self-host OpenMapTiles but gave up and used Stadia Maps since it would have taken over 100 days [1] to generate a map of the world myself. Stadia Maps has been great and I highly recommend them, but the problem continued to nag me, so earlier this year I started prototyping a new way to generate vector tiles faster. The idea worked and ended up being able to generate a ~100GB planet.mbtiles file in as little as 59 minutes on a single c5ad.16xlarge instance with 64 cpus and 128GB RAM (although smaller machines appear to be more economical).

I’m open-sourcing Flatmap today. Let me know what you think, or if you have any suggestions!

[1] https://github.com/openmaptiles/openmaptiles/issues/654#issu...