Replacing Mapbox with open source solutions

https://www.kschaul.com/post/2023/02/16/how-the-post-is-replacing-mapbox-with-open-source-solutions/

by gaws • 3 years ago

613 112 3 years ago

ecf• 3 years ago

Currently using Mapbox in an open-source project for their static map API and I wholeheartedly recommend against it. Everything but the simplest of polylines look jagged since they refuse to switch to POST requests with a json body and instead rely on you fitting in as much detail into URL query parameter length constraints.

Going to take a look at replacing them with PMTiles and my own polyline rendering.

leetrout• 3 years ago

Subscribing for updates :)

Will you be talking about this work openly somewhere?

andy_ppp• 3 years ago

Are there decent label placement solutions these days? It’s a really tricky problem to do well.

tylershuster2• 3 years ago

I've switched over a fairly large transit client to PMTiles hosted on CloudFlare for their dynamic map and am switching another large transit client to using https://github.com/maptiler/tileserver-gl with MBTiles and the same style.json to generate static maps. It's all gone swimmingly and we've cut costs by about $1,000 a month.

lukeqsee23• 3 years ago

There's actually a reason why most of these APIs support POST—most use-cases are hot-linking, where you don't control how the client makes the request. (Often using an <img>.)

Here's one alternative, if you don't want to build from scratch: https://docs.stadiamaps.com/static-maps/ We don't have much in the way of query string limits, so have at it!

capitainenemo• 3 years ago

Rather curious about the URL query parameter length constraint. Excluding IE11/Edge, don't most major browsers have at least 64k query lengths these days? Could you just serve the low quality jaggy poly to few percent still on IE11/Edge and a high detail one to the rest?

lukeqsee23• 3 years ago

A lot of servers impose a constraint on URL length, so I suspect Mapbox does on their APIs.

danpat• 3 years ago

You always have the option of creating a map style and using Mapbox's Raster Tile API - although that does mean you probably can't dynamically adjust your raster images in time for a request to be served.

How are you using the Static Maps API currently where you would be able to send a POST request and display the image back to the user? Most use-cases for the Static Maps API involve the HTML <img> tag or something similar, which all send GET requests.

tpmx2• 3 years ago

Wasn't Mapbox once the low-cost option?

Operyl• 3 years ago

Yup, which disrupted the market, and now it's not.

klysm• 3 years ago

1. Subsidize low costs with venture capital and take on a ton of customers at a loss 2. Lock those customers into your API somehow. Enough code is effectively lock in for a while. 3. Change your pricing to be profitable. 4. Hopefully get acquired before all your customers leave

gorbypark2• 3 years ago

They used to be low-cost _and_ open source. Now they're source available and not interested in anything but big clients. I was using Mapbox for all my clients before their unopen sourcing, and I tried reaching out three times to ask about options for one of my clients. It wouldn't have been a huge amount of money, probably a few hundred per month, but they never, ever reply. That client wasn't even using any of their server resources, they are 100% self hosted, it's just that Mapbox now charges even for using their client. Anyways, I was trying to give them money, and they wouldn't even acknowledge I exist. I switched that client over to Maplibre and never looked back.

The other thing that annoys me is that they've more or less abandoned all their open source repos without even as much of a notice in the README. All these amazing tools sitting there with open pull requests and no one responding for years. If they just did a clean break, put a notice on everything they weren't going to touch again, it would have been a lot better/easier for the community to fork and move on with their lives.

shawn-butler• 3 years ago

Mapbox was born from open source tools & culture. They were always active and supporting at FOSS4G.

I wonder what has changed?

usrusr2• 3 years ago

And since the state of the open source map rendering art is consisting almost exclusively of pre-licence-change forks of mapbox code these days, they kind of still are.

I wonder if Strava's recent acquisition of fatmap.com eventually boils down to them cutting ties with mapbox via acqui-hiring some mb-libre excellence. I could imagine Strava, due to the extreme map-centric nature of their product, bleeding quite considerable money to mapbox.

lukeqsee23• 3 years ago

MapLibre has made really good strides (especially on the web) since the fork—3d-terrain support, more tile sources, other DX improvements (e.g., conversion to TypeScript).

https://maplibre.org/maplibre-gl-js-docs/api/

kylebarron• 3 years ago

I didn't think Fatmap was based on mapbox at all? I thought it was their own custom renderer

jeffbee• 3 years ago

The Post has been doing great data journalism lately. The maps from the article were really well done. Yesterday they ran this story that has all its graphics done in D3 and provides links to notebooks at ObservableHQ with evidence supporting the arguments made and conclusions drawn in the article. https://www.washingtonpost.com/climate-environment/interacti...

jackson1442• 3 years ago

Looks great. Been using Mapbox as my go-to for small projects since it's much smoother than Leaflet and has better DX in my opinion. This seems to capture all of that with an even better DX so definitely will be trying it out.

lukeqsee23• 3 years ago

https://maplibre.org/ definitely attempts to preserve the DX with free selection of tile providers.

tacker2000• 3 years ago

DX == Developer Experience? Thats a new one.

jokoon• 3 years ago

I don't want to sound annoying, but it's a bit hard to generate raster tiles. I mean openstreetmap data is open and free, but it doesn't seem they distribute any software to build the raster tiles they host.

I don't know what they use, but it doesn't seem there is any software that is simple enough to use.

I just want to configure a subregion, call generate-tiles.py planet.xml or whatever, and be done with it.

I like open source, but there will always be open source software that is needlessly complicated because, well, it lets companies like mapbox and others to generate income for their expertise.

That's not the best open source spirit, in my view.

livinglist• 3 years ago

Test

uneekname• 3 years ago

Open-source geospatial tools in general have developed rapidly over the last couple of decades. Mapbox and friends have laid groundwork, both good and "needlessly complicated." But there are just too many people who care about this to think it'll always be hard.

I assure you, the OSS (and in this case, OSM) community will steadily build out better, faster software that includes more and more of the features a modern map needs. It's already looking like companies like WashPo are seeing the potential here.

Come join us and make the world a better place :)

tylershuster2• 3 years ago

https://github.com/maptiler/tileserver-gl makes it really easy. I have a pretty simple pipeline set up to download an OSM extract, convert it to MBTiles, host it in CloudFlare, and render raster tiles with this software.

lukeqsee23• 3 years ago

Checkout https://github.com/onthegomap/planetiler.

Super easy way to generate a MBTiles, which you can then serve directly, or further convert to PMTiles, which can be used to host vector tiles for client-side rendering using MapLibre (or other renderers).

Raster tiles are a lot harder because you have to generate them on the server, and that's a lot more resource intensive.

j_heffe• 3 years ago

We used https://github.com/developmentseed/titiler (and rio-tiler, the underlying library) at my last company which does dynamic tiling based on some input raster. It's an awesome project and there's nothing else quite like it in the geospatial space.

lukeqsee23• 3 years ago

I discovered it while looking at raster solutions, and it is pretty great! The raster stack is pretty bonkers. TiTiler uses at least three fundamental layers: GDAL, RasterIO, and rio-tiler. Each contribute a fairly significant improvement (for the use-case) over the underlying one.

Doctor_Fegg2• 3 years ago

The classic instructions for raster tiles are at https://switch2osm.org. I think there’s a Docker image if that floats your boat.

KronisLV• 3 years ago

These are some really great instructions, I wish other sites would be so informative and approachable.

They do have more details on the Docker container approach in one of the pages as well: https://switch2osm.org/serving-tiles/using-a-docker-containe...

Here's the relevant Docker Hub link: https://hub.docker.com/r/overv/openstreetmap-tile-server

The requirements are considerable, though, given the workload:

Serving your own maps is a fairly intensive task. Depending on the size of the area you’re interested in serving and the traffic you expect the system requirements will vary. In general, requirements will range from 10-20GB of storage, 4GB of memory, and a modern dual-core processor for a city-sized region to 1TB of fast storage, 24GB of memory, and a quad-core processor for the entire planet.

Personally, for lower end mobile devices raster tiles still seem to perform better, when compared with at least some of the current vector tile implementations. I actually recorded a short comparison on my phone a while back: https://www.youtube.com/watch?v=A-IRtBGO9rM

lukeqsee23• 3 years ago

This is really awesome to see.

I'm involved in many of these projects (helped get MapLibre started, wrote a library supporting PMTiles, have used OpenMapTiles for years), and to see open-source, open-data win in the end is really satisfying.

If anyone has any questions about how you could do this for yourself, I'm happy to answer! (I have a company offering SaaS maps, but we know SaaS isn't for everyone, and are happy to point people the right direction when self-hosting is what you want.)

jddj3• 3 years ago

Is there a standard folder tree of tiles -> PMTiles archiver or is that the wrong question to be asking?

lukeqsee23• 3 years ago

I don't think so. There's this: https://github.com/protomaps/go-pmtiles to get from MBTiles and there's a few xyz folder tree tools to get mbtiles from that structure (e.g., https://pypi.org/project/mbutil/). So you can get there, but I don't know of a good straight-line right now.

bdon4• 3 years ago

There isn't a tool to do that right now. It could be a fit in either https://github.com/protomaps/go-pmtiles or https://github.com/protomaps/PMTiles/tree/main/python - the Go program is faster and more production ready at this point. I imagine if folders are working for you the quantity of tiles doesn't number into the millions, so the Python program might be sufficient.

Feel free to open an issue.

franga20002• 3 years ago

Thanks for all your work on MapLibre, along with all the other contributors! I use MapLibre extensively in an open source and non-profit transit data aggregator and everyone keeps being blown away by how smoothly the map runs, even with tens of thousands of points.

We burn through MapTiler free tiles on a regular basis and have been looking to migrate to self-hosting, but somehow PMTiles flew under my radar and every other approach seemed to either still rely on MapTiler for data (where you have to chose between outdated or expensive) or just be a huge pain to set up. Hopefully this finally solves our map issues so we can actually start promoting the project without fear of the map disappearing in the first week every month.

lukeqsee23• 3 years ago

My company may be able to help save you the trouble of self-hosting. We do have a free tier (for non-commercial / test usage), and our pricing structure allows you to save quite a bit compared to MapTiler.

762236• 3 years ago

If all of the tiles are coming from a single file (via PMTiles), how are they cached by the client? Do clients cache byte ranges, or do you configure things to give the appearance of serving separate files?

lukeqsee23• 3 years ago

You typically use byte-range caching in libraries—either using an edge worker (a la Cloudflare) to create separate GET requests from the client's perspective or directly in the client.

cancan2• 3 years ago

i think you can also directly call them from the maplibre lib itself using the pmtiles adapter

https://github.com/protomaps/PMTiles

https://github.com/protomaps/PMTiles/blob/main/js/examples/m...

bdon4• 3 years ago

The renderer used in the linked article (MapLibre) has an internal tile cache used when you return to a previously loaded area of the map.

For the non-tiled parts of the PMTiles file, like metadata and the directories that store Z/X/Y to offset information, those are LRU cached by the PMTiles implementation. The most widely deployed right now is for TypeScript/Node. The "client" can either be web browser or inside a runtime like Lambda/Cloudflare Workers. On those the cache will persist across invocations using the ephemeral memory of the serverless function.

A level deeper, browsers vary in how they cache Range requests; IIRC Firefox does not cache while others will treat Range as if it was any other HTTP header.

kabes• 3 years ago

Worth mentioning that openlayers is also really extending their webgl rendering capabilities lately.

lukeqsee23• 3 years ago

There's also support for MapLibre GL JS style specs in the works! Excited to see another stable vector tile & style renderer added to the ecosystem.

wipfli• 3 years ago

If you like PMTiles you should consider supporting Brandon financially as a GitHub Sponsor.

Brandon is a solo-developer who builds Protomaps alone and we get all his great stuff for free...

https://github.com/sponsors/protomaps

lukeqsee23• 3 years ago

You can also sponsor MapLibre: https://github.com/sponsors/maplibre

(Disclaimer: I'm on the governing board, but derive no financial benefit from it.)

pella• 3 years ago

And https://donate.openstreetmap.org/

"OpenStreetMap is the largest open geographic database in the world, the data infrastructure for multitudes of mapping projects around the globe. Your donation to the OpenStreetMap Foundation will cover our core operational expenses in supporting the OpenStreetMap project: hardware costs, legal fees, administrative assistant and other expenses of our working groups and administration.

We currently run extremely lean for an operation for a project the size and importance of OpenStreetMap. The OpenStreetMap Foundation relies on revenue from individual and corporate membership dues, profits generated by the annual State of the Map conferences, and donations. We must keep our income sources diversified, as these vary from year to year, but our modest needs stay the same. For this, we need your support."

timwis• 3 years ago

Also see valhalla for an open source routing engine: https://github.com/valhalla/valhalla/

lukeqsee23• 3 years ago

Valhalla is great (I've contributed a few patches and run it for a while)!

Also check:

http://project-osrm.org/

https://www.graphhopper.com/open-source/

notyourday• 3 years ago

Open source use we will, pay for development we won't, paywall for our "reporting" we have

ARandomerDude• 3 years ago

Write like Yoda you do.

margorczynski• 3 years ago

That's why I actually support paid licensing for commercial use. I'm all for helping out people but not a fan of getting taken advantage off - you make money off my or someones else's work then pay up.

0xDEF• 3 years ago

Leaflet is also an interesting solution in this space:

https://github.com/Leaflet/Leaflet

aendruk• 3 years ago

Vector tiles?

lukeqsee23• 3 years ago

https://github.com/maplibre/maplibre-gl-leaflet exists!

It's not perfect, and you don't see the full benefit of a WebGL renderer, but if you want to keep using a Leaflet API, it's great.

lights0123• 3 years ago

Although I used to exclusively use it, Canvas-based maps just run so much smoother and support things like rotation that are really helpful for changing perspective.

flakeoil• 3 years ago

It's weird, 10 years ago I implemented using Google Maps API a website with dynamic maps, distance matrix, geolocation, map overlays and address autocomplete completely free also at high traffic. It was also easy to use.

Five years ago Google started to charge an outrageous amount > $1000/month and we could not continue so we switched to something that cost $100/month. And that was for only the maps. We had to scrap the distance matrix and address autocomplete.

Today it's still the same status, and if you want something cheaper then you have to work on putting pieces together like the Washington Post does.

lukeqsee23• 3 years ago

We offer a very competitive solution that is quickly becoming a one-stop shop for what you mentioned: https://stadiamaps.com.

All that functionality for a lot less than $1000 / month.

countvonbalzac• 3 years ago

you should put a price comparison for stadia vs. google vs. mapbox

notatoad• 3 years ago

and as a bonus you've got alidade smooth, the ideal map tile for overlaying colorful data on top of: https://stadiamaps.com/themes/

vinibrito• 3 years ago

Hey guys not affiliated with Stadia but I can vouch for them, I use it in one of my products for a few years now and never heard a complaint, only nice things. You can't go wrong with them.

keltex• 3 years ago

Same here with our website https://www.meetways.com. This product was created around the time google created their maps API. In fact we worked extensively with google to help them iron out some of their API issues. They even discuss our product on their blog: https://mapsplatform.googleblog.com/2008/10/geocoding-in-rev...

This site made a little bit of money, but not that much. But then 5 years ago google raised their prices so much that we were suddenly in the red. We had to drop the places API (replaced with Yelp) and the autocomplete API (we don't even use that anymore). Recently we've been moving to Apple Maps API which has a pretty generous free tier. Stadia Maps also looks interesting.

lukeqsee23• 3 years ago

Would love to chat to see if we can help you! If you want, give me a shout at luke <at> stadiamaps.com

folli2• 3 years ago

I'm not affiliated, but a reasonably satisfied customer of https://maptiler.com for my project https://cubetrek.com. Perhaps also worth to take a look.

Reason077• 3 years ago

”Stadia Maps”

Interesting name, considering Stadia was the brand of a recently-discontinued cloud gaming platform made by… Google.

trynewideas• 3 years ago

In the tools space, cf. open source GeoJSON-to-vector tile tool Tippecanoe, which recently got a new non-Mapbox home at Felt: https://felt.com/blog/erica-fischer-tippecanoe-at-felt, https://github.com/felt/tippecanoe

Doctor_Fegg2• 3 years ago

But for most of our use cases, we don’t need the latest and greatest.

This is the crux of this. Let’s not pretend that much of this is anything other than “open source clones of Mapbox”.

PMTiles is genuinely innovative and I’m delighted to see it getting traction.

Maplibre GL (which I use and like very much) is a fork of the last open source version of Mapbox GL, with not many code changes between now and then.

Maputnik is an open source reimplementation of Mapbox Studio. Mapbox went on to hire its original dev.

OpenMapTiles started life as basically a how-to/makefile pulling together the disparate, poorly documented parts of Mapbox’s open source tile generation code. It’s now mostly known for a reasonable, not particularly exceptional vector tile schema.

It’s great that people are now shipping production vector tiles without paying the Mapbox dollar. But, PMTiles aside, this is still Mapbox’s world. The next challenge is to evolve the tech stack to something beyond what Mapbox worked up five/ten years ago.

lukeqsee23• 3 years ago

PMTiles aside, this is still Mapbox’s world.

Isn't this a bit like saying we're still in Google's world because most maps still use Web Mercator? :)

Good tech builds on what came before. Mapbox did a lot of ground-breaking work in building tooling around OSM, but so have many others. The fact that they named it Mapbox Vector Tiles is genius in hindsight, because even though we may use tons of tooling they didn't create to build and render them, their name is still there.

The next challenge is to evolve the tech stack to something beyond what Mapbox worked up five/ten years ago.

Agreed, and I think we've seen a lot of iterative work in the open since then. The next challenge is likely building a OSS stack to do proper 3D: open data (including OSM) to pixels, and that work is already beginning across a lot of organizations: https://github.com/nyurik/future-mvt/discussions, Overture Maps, MapLibre, etc.

twelvechairs• 3 years ago

I get the parent comment. There are good alteratives to the mapbox ecosystem though they arent always used because people see mapbox as default

DeckGL for instance is a very innovative js library for front end (really developed by Uber I believe), with better 3d and movement, better with dealing data at scale, etc.

Prettymaps is a pure python approach to rendering OSM data.

Etc.

cancan2• 3 years ago

[founder of felt, maintainer of some of the libraries mentioned]

this is amazing to see! we are happy to see our OS contributions like tippecanoe are being used by people all over the place, and i personally really love to work with journalists.

happy to answer any questions!

lukeqsee23• 3 years ago

Thanks for taking up the maintenance of Tippecanoe! I haven't needed to use it too much, but when I have, it fills a nice void.

sgt• 3 years ago

Geoserver is also good for serving spatial data and tiles. A bit dated look but fast and stable

wiredfool2• 3 years ago

I find geoserver to be a bit of a headache to run, and have been moving more towards client side rendering using map ox style tools.

guwop3• 3 years ago

Do you know of any good alternatives to geoserver if i have a PostGIS db with some feature gis data and i need to expose it to other people through a Web Feature Service?

preya2k• 3 years ago

Is there an OSS way to use vector tiles, without using Mapbox?

mtmail• 3 years ago

The article covers that. See the "maplibre-gl-js for client-side rendering" section.

rhodysurf• 3 years ago

Yes, just use tippecanoe to create your tiles, and serve them up via s3 or howerver else you want. Then client side render them with maplibregl

Demiurge• 3 years ago

Check this out for backend: https://postgis.net/docs/ST_AsMVT.html if you can run ogr2ogr to import your data into PostGIS, and serve the binary data returned by a query, you're all set. You literally only need PostGIS for a dynamic vector tile service.

Then mapboxgl or libremapjs on the frontend, or even OpenLayers.

guwop3• 3 years ago

So to get it straight:

OSM.pbf file -> ogr2ogr -> PostGIS?

I currently use a .mbtiles file and plug it into tileserver-gl, but i guess in theory you could just import OSM to PostGIS and then add the PostGIS as a source in tileserver-gl? That would be amazing

didip• 3 years ago

Is Mapbox dead now? It used to be awesome many years ago.

__turbobrew__2• 3 years ago

Titiler works pretty well for raster data in my experience. It is a step up from using an out of the box solution and involves a bit more development work but pretty easily allows customization. Using titiler on lambda with COGs stored on S3 we could get about 200ms load of a retina resolution map.

We wanted to do things even faster so we started migrating to ECS which when using a network optimized instance we could render a full page map in sub 100ms.

The author states that having all the data in a single file for PMTiles is advantageous, but it is a tradeoff. First of all you cannot modify parts of a S3 file so if you want to modify/update your dataset you need to re-process and re-upload the entire dataset. This can be a real pain if you are working with many TB datasets and you just want to update a subset of the data. Also, I thought that there is a global throughput limit on S3 files so wouldn’t putting all data in a single S3 file potentially hit bottlenecks?

bdon4• 3 years ago

Yes, PMTiles is a tradeoff that isn't appropriate for transactional use cases. SQLite is pretty good for that already.

There is a throughput limit on S3 files of approximately 5500 GETs/sec per key. Bare archives on S3 is an appropriate choice for small-scale, zero maintenance deployments. If your application demands any thing close to that level of throughput, you're probably either:

* Serving individual tiles over the internet: you should use the CDN integration http://protomaps.com/docs/cdn ; most tile requests will be cached and only misses will interact with the S3 bottleneck.

* Bulk accessing a spatial subset of tiles: You shouldn't be requesting HTTP GETs for single tiles, but instead entire subsets of tiles with a single Range request made possible by the internal Hilbert curve ordering. This is still WIP here: https://github.com/protomaps/go-pmtiles/issues/31

neuronexmachina3• 3 years ago

Using titiler on lambda with COGs stored on S3 we could get about 200ms load of a retina resolution map.

Since Lambda presumably has a separate GDAL RAM cache for each request, I'm guessing this wasn't able to take advantage of any of GDAL's header or block caching? It's pretty impressive you were able to get latencies that low with that approach.

__turbobrew__2• 3 years ago

The big performance gains of GDAL in this configuration is HTTP2 multiplexing to S3 and TCP keepalives. When a tile request is sent to the lambda tiler it should already have a TCP/TLS connection open to S3 which can then be used to multiplex requests.

pierotofy• 3 years ago

OpenMapTiles is open source just as much as OpenAI is open source.

goplayoutside• 3 years ago

While it might be nice if there we pre-rendered tiles available for download, and openmaptiles.org constantly steers users to "buy now" pages, instead of offering a simple button to the self-service docs, they do in fact offer a significant amount of documentation on how to generate and serve your own tiles[1], including a page with fairly step-by-step instructions on how to create a tileset "from scratch" without sending them a cent[2].

1. https://openmaptiles.org/docs/ 2. https://openmaptiles.org/docs/generate/generate-openmaptiles...

time4tea• 3 years ago

I think you can do this quite easily. Use docker compose to start an nginx and maptiler/tileserver-gl. Use an nginx config to send requests upstream to tileserver and cache.

Download a region file from maptiler, or make one.

That's it.

It's about 2h work, using certbot for certificates.

If you want to create your own styles, it's slightly more fiddly, but essentially it's https://maputnik.github.io/editor/#0.41/0/0

Host on hetzner for €3/month.

jillesvangurp• 3 years ago

We use maplibre and map tiler for the map tiles in the startup that I run. We switched to that from leaflet and I've been pretty happy with it. I might invest some time at some point in self hosting tiles. But honestly, map tiler is good value for money for now.

I actually used to work with Nokia and Nokia Maps (in Berlin), which is now Here Maps. So, I know a thing or two about this market and still know some people working with various map companies, including at mapbox.

Mapbox is increasingly an alternative to Here, Tom Tom and a few others that offer a wide range of services. The most simple one of which is showing maps. Think of indoor maps, traffic services, routing, geo coding, address search (surprisingly hard problem with open data), etc. Customers for these services tend to spend a lot of money on getting all sorts of bespoke solutions. Maps are a commodity at this point and there's very little money in providing just some basic maps. It's all the other stuff that's built on top of that that is a lot harder and more valuable.

The recently announced overture collaboration could actually shake things up a little in this space. Openstreetmap is nice but also has some pretty significant limitations. Basically what the above mentioned companies do is creating proprietary services and data (with and without open data typically) that they then sell access to for a lot of money. The big companies behind overture are each big customers for this stuff. And now they are pooling resources to create a cheaper, more open alternative to all that.

mapnostic• 3 years ago

This discussion is great to see! We’ve been talking about this article at work, the only thing I’d point out that hasn’t been said is that there is some complexity to designing with and managing the other renderer assets besides tiles (i.e. styles, markers, textures, icons, fonts). Most of this complexity is well documented if you know where to look, but will trip you up at least the first time you set up a serverless mapping stack, so buyer beware.

The trickiest of these assets is the stylesheet, necessary for the renderer to know how to draw the data in tiles. The article mentions Maputnik, which works great for quick prototyping and the production deadlines of newsrooms, but limiting once your map, application, and or tiles get to a certain level of complexity.

Replacing Mapbox with open source solutions

Related Stories