Overture Maps Foundation releases open map dataset

https://overturemaps.org/overture-maps-foundation-releases-first-world-wide-open-map-dataset/

by chippy • 3 years ago

402 100 3 years ago

pininja2• 3 years ago

POIs, buildings, transportation network, and admin boundaries layers.. on my!

dhx2• 3 years ago

I have an interest in this topic as a contributor to AllThePlaces[1], an open source project collating Scrapy spiders (MIT license) that crawl websites of franchises/retail chains that you'd find listed in name-suggestion-index[2] to retrieve location data (CC-0 license). The project is just short of collecting 3 million points of interest from almost 1700 spiders.

Overture Maps appears to be quite a closed and proprietary project, with claims of openness limited to being able to download a data set and accompanying schema specification. Some issues that immediately come to mind:

1. There is no published description for how the data was generated. End users thus are given no assurance of how accurate and complete the data is.

a. As an example, administrative boundaries are frightfully complex and include disputed boundaries, significant ambiguity in definition of boundaries, and trade-off between precision of boundaries versus performance of algorithms using administrative boundary data. Which definition of a boundary does Overture Maps adhere to, or can it support multiple definitions?

b. It's probable that Microsoft have contributed ld+json/microdata geographic data from BingBot crawls of the Internet. This data is notoriously incorrect, including fields mixed up and invalidly repurposed, "CLOSED" in field names to denote closure of a place 5 years ago but the web page remains online, and much ambiguity in opening hours specifications. For AllThePlaces, many of the spiders developed require human consideration, sometimes of considerable complexity, to piece together horribly messy data that is published by shop and restaurant franchises, and other organisations providing location data via their websites.

c. For location information where +/- 1-5m accuracy and precision may be required (e.g. individual shops within a shopping centre[3]), source data is typically provided by the authoritative sources with 1mm precision and +/- 10-100m accuracy. AllThePlaces, Overture Maps, Google Maps and similar still need human editors (OpenStreetMap editors) to do on-the-ground surveys to pinpoint precise locations and to standardise the definition of a location (e.g. for a point, should it be the centroid of the largest regular polygon which could be placed in the overall irregular polygon, the center of mass of a planar lamina, the location of the main entrance, or some other definition?).

d. If Overture Maps is dependent on BingBot for place data, they'll miss an enormous number of points of interest that BingBot would never be able to find. For example, an undocumented REST/JSON/GraphQL API call or modification to parameters to an observed store locator API call may be necessary to return all locations and relevant fields of data. Website developers routinely do stupid things with robots.txt such as instruct a bot to crawl 10k pages (1GB+) from a sitemap last updated 5 years ago rather than make 10 fast API calls for up-to-date data (5MB). Overture Maps would be free to consume data from AllThePlaces as it is CC-0 licensed, and possibly correlate it with other data sources such as BingBot crawl data, a government database of licensed commercial premises or postal address geocoding data. However the messiness of data in various sources would be approaching impossible to reconcile, even for humans, and Overture Maps would possibly have to decide whether to err on the side of having duplicates, or lack completeness.

2. There is no published tooling for how someone else can reproduce the same data.

a. AllThePlaces users fairly frequently experience the wrath of Cloudflare, Imperva and other Internet-breaking third parties, as well as custom geographic blocking schemes and more rarely, overzealous rate limiting mechanisms. If Overture Maps is dependent on BingBot crawls, they'll have a slight advantage over AllThePlaces due to deliberate whitelisting of BingBot from the likes of Cloudflare, Imperva, customer firewalls, etc. However, no matter whether you're AllThePlaces or Overture Maps or anyone else, if you want to capture as many points of interest as possible across the world, use of residential ISP subnets and anti-bot-detection software is increasingly required. They'll need people in dozens of countries each crawling websites targeted to the same country, using residential ISP address space. Otherwise they end up with an American view of the world, or a European view of the world, or something else that isn't the full picture.

b. If Overture Maps has locations incorrect for a franchise/brand due to a data cleansing problem or sourcing data from a bad source (perhaps non-authoritative), there are no software repositories for the franchise/brand to raise an issue or submit a patch against.

[1] https://www.alltheplaces.xyz/

[2] http://nsi.guide/

[3] Example Australian shopping centre as captured by AllThePlaces: https://www.alltheplaces.xyz/map/#18.07/-33.834646/150.98952...

mistrial9• 3 years ago

strong agree -- additionally, there is clear public relations going on, announcing maps like this as "trusted" or "official" backing, presumably for decisions over commerce, safety, insurance claims or other non-security but valuable, data-oriented activity. The public or public oversight, has what recourse here?

tony_cannistra5• 3 years ago

2. There is no published tooling for how someone else can reproduce the same data.

This is true, and is actually the whole point of Overture.

Overture was developed to enable private companies to leverage open data (like OpenStreetMap) but also combine it with their proprietary data and processes.

The intention is to share the result with a relatively permissive license (a new thing called the Community Database License Agreement) but keep the process and underlying data proprietary.

justinclift2• 3 years ago

Sounds like that's going to be a problem for proving/reproducing results independently then. :(

uneekname5• 3 years ago

Yes, exactly. Overture is, for better or worse, looking to create a dataset that is more difficult to mess with / contribute to.

I'm big on OpenStreetMap, but I can't deny its a bit of a liability for Facebook and other companies that display maps to their users. There is the occasional vandalism edit that simply can't be shown to an end-user. Facebook put significant effort into maintaining a moderated version of the OSM database that lags behind the real-time edits. Facebook, Microsoft, TomTom, etc. know this is a ton of work and want to pool their resources. Making it open also helps to openly compete with Google, the other big map data provider.

If you want to contribute to Overture as an end-user, AFAICT your best option is to edit OpenStreetMap and see if your changes eventually get pulled in. Overture has promised the OSM community that they'll make much of their data available to be contributed back, we'll see if that pans out.

When it comes to AllThePlaces -- as an OSM nerd it seems like there is an opportunity to build a better bridge between this and OpenStreetMap, to make it easier to quickly update businesses in an area. Recently there has been a pretty successful push to link OSM data with WikiData, using tools like the OSM ↔ Wikidata matcher [0]. For POIs, it's a lot of work to add a bunch of local businesses, even with tools like EveryDoor [1]. It would be so cool to see AllThePlaces integration into RapID for example, if there isn't already(?)

[0] https://osm.wikidata.link/

[1] https://every-door.app/

stderrout• 3 years ago

Wonder what could be reasoning behind this. Is it they dont want disclose data collection practices or cant do from legal points.

Things like it could come from alexa, pc or any devices devices with forced opt ins that keep scanning all your neighbours wifi networks and mac addresses.

Believe there was also an initiative where amazon devices will provide adhoc internet connectivity by piggy backing on other amazon devices on different networks with connectivity.

So all the openness but without any controls. There should already be a better term for things like this.

dhx2• 3 years ago

Wouldn't the closed processes and underlying data severely limit communities such as OSM from using Overture Maps results for anything other than a validation of what OSM already knows from other sources?

Perhaps Overture Maps has used impressively accurate satellite imagery tracing to detect the demolition and rebuild of a structure somewhere in Sudan, and can output a new polygon. No OSM mapper is setting foot in Sudan, and recent satellite imagery for the area is not available through companies that share such data for OSM use.

The issue for an OSM mapper who sees the conflict between OSM (with the old building) and Overture Maps (with the new building) is they don't have any information to know which result is accurate. Is OSM just out of date? Has Overture Maps produced the result from outdated satellite imagery and OSM is more up-to-date? Is the result form Overture Maps the result of a mistake in an automated tracing algorithm?

mikeyouse3• 3 years ago

The GitHub repo is here; https://github.com/OvertureMaps/data -- Licenses look super permissive - I'm not that familiar with the state of Open Mapping but the GERS idea looks great too.

https://docs.overturemaps.org/gers/

orra• 3 years ago

The CDLA license feels like NIH... but at least it looks genuinely permissive, as opposed to fake open source.

Curiously, two of the four layers use the ODbL.

matkoniecz7• 3 years ago

Curiously, two of the four layers use the ODbL.

Probably this data is reformatted OpenStreetMap data or derived from it (so is licensed as ODBL and requires attributing OpenStreetMap)

sdfghswe• 3 years ago

What's the difference between this and Open Street Maps?

pininja2• 3 years ago

To start, the admin boundaries and POIs are given a more permissive license. They’re mission is to make permissive map datasets.

mikeyouse3• 3 years ago

Ha I was just wondering the same and Overture has a FAQ that answers that exact question;

Overture is a data-centric map project, not a community of individual map editors. Therefore, Overture is intended to be complementary to OSM. We combine OSM with other sources to produce new open map data sets. Overture data will be available for use by the OpenStreetMap community under compatible open data licenses. Overture members are encouraged to contribute to OSM directly.

zigzag312• 3 years ago

We've focused on designing a data schema that is easy for developers to quickly understand and use in building map products.

https://docs.overturemaps.org/

OSM focuses on open map editing, but due to it's flexible schema, it can be hard to extract needed information from OSM.

Overture seems to focus on providing map data (from multiple sources) that can be used more easily.

EDIT: OSM is also trying to improve the OSM data model for easier processing. https://github.com/osmlab/osm-data-model

matkoniecz7• 3 years ago

OpenStreetMap is community of people creating fully open map database. From roads, shops and rivers to tourism attractions, hiking routes and hospitals, with all kind of detail. Anyone can contribute (and as someone quite involved in OSM: if you want to map something, especially area near you: you are welcome!).

Overture is a new group of companies releasing some datasets on open licenses, but methods used to create them remain proprietary. Some of released data is theirs, many datasets are repackaged OpenStreetMap data.

leokennis• 3 years ago

In some ways OSM is more impressive than Wikipedia.

There’s a gravel path near my house that maybe sees 20 people using it daily. Due to some work done nearby, the path was partially moved a few meters to the side. OSM reflected this new reality the day after.

hk__22• 3 years ago

This really depends on the density of contributors. In Paris every single tree is mapped on OSM, while in more remote areas it’s not rare to see entire roads or even villages missing.

Doctor_Fegg3• 3 years ago

OSM is predominantly human-made. This is predominantly machine-made.

hk__22• 3 years ago

Machine-made …from OSM and other sources.

izacus• 3 years ago

This license-washes Open Street Maps data into proprietary systems so big corps can use it without contributing back.

tony_cannistra5• 3 years ago

For context, Overture Maps is a project that intends to enable large players in the geospatial space (TomTom, Amazon, Microsoft, but notably _not_ Google) to leverage open data sets (OpenStreetMap among them) alongside proprietary data and processes that they own.

The consortium intends to enable a framework for enhancing geospatial data based in open data sets (e.g. OSM) with their own proprietary processes, and re-release it with a permissive license (the Community Database License Agreement - CDLAv2), but keep the data and processes required to create that dataset proprietary.

The project has created a lot of conversation in the OpenStreetMap community, but in general I think it's good to see so many resources put into the OSM-adjacent world.

gumby• 3 years ago

I came to ask for this info so thanks for the succinct summary!

alexchamberlain• 3 years ago

How do they do that? I thought OSM was licensed in such a way that if you combine it with other data, the whole thing needed to be released with the same license?

tony_cannistra5• 3 years ago

You are mostly right, except that in addition to the same license, the Share-Alike provision in the OSM license (the ODbL [0]) also enables derivatives to be re-released with "A compatible license."

I am not a lawyer, but I would assume someone at Overture has deemed their license (the CDLAv2 [1]) to be "compatible" with ODbL. (edit: From the link in OP, the OSM-based data is being released with ODbL. I think I'm wrong about license compatibility. Again, not a lawyer.).

[0] https://opendatacommons.org/licenses/odbl/1-0/ [1]: https://cdla.dev/permissive-2-0/

callalex• 3 years ago

These things don’t matter when one side has 100x the lawyers of the other side.

matkoniecz7• 3 years ago

Are they releasing OSM data or OSM derived data on CDLAv2 anywhere?

RicoElectrico2• 3 years ago

Another bit of context: OSM mappers were quite discontent with edits of many of Overture members. Be it low-paid contractors or bulk importing stuff that AI derived from satellite imagery.

grishka2• 3 years ago

bulk importing stuff that AI derived from satellite imagery.

That's a thing? This must explain why Google Maps turned into shit over the last few years. Google seems obsessed with replacing humans with half-assed automation.

culi2• 3 years ago

Do you know when these contributions began? Has the scene changed much since then?

claytongulick• 3 years ago

It would be super helpful if they at least released the name of their data sources.

The places layer would be invaluable for (for example, stuff I'm working on now) geospatial healthcare access analytics if I could rely on the places being a reasonably accurate source of provider locations.

It's a lot of work to tie all the CMS data with health plan data etc.. and then to geocode it all with OSM tiger geocoders. If I could rely on that data already being present and scrubbed, it would be a godsend.

jmye• 3 years ago

Your project is really interesting to me - are you just looking at the physical buildings (e.g. you have 5 urgent cares within X distance), or are you trying to cross-reference provider data to number doctors/specialties? Or something else entirely?

Totally understand if you can’t discuss (or don’t want to), but it’s in my domain and I’ve spent a fair amount of time thinking about how to do it without coming up with much, so I’m super curious about your thoughts.

joshstrange4• 3 years ago

If you are looking for maps themselves (a la Google Maps) I highly recommend checking out Protomaps [0]. It provides a single file (PMTile format) that contains all the map data and you can request ranges from the single file to get the required vector data you need for a given area/zoom.

So you can setup something like the following:

    S3 (host the myfile.pmtiles) <- Lambda (takes the x/y/z from the path and requests the correct range) <- Cloudfront cache tile response

Then you can setup tiles.mydomain.com (or use the cloudfront domain directly) and then use Leaflet or similar on the frontend to fetch/render the tiles. For Leaflet you use the protomaps plugin/lib and give it a url like "https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt" where "20230408" maps to "20230408.pmtiles" in your S3 bucket. Now I can drop new pmtiles files into that bucket and update my clients to use the new source. And since the tiles are in vector format you can theme them however you want in the client which is neat. Lastly you don't have to use the 100+GB whole-earth tileset. You can use a tool [1] (provided by the same guy) to download a dataset for just a given geographical region.

The .pmtiles file is a little over 100GB but the whole setup took me only an hour or two max to get running and will cost way less than Google Maps to run.

[0] https://protomaps.com/

[1] https://app.protomaps.com/downloads/small_map

mdaniel5• 3 years ago

their demo isn't doing them any favors since it seems to stop at some random zoomlevel, eliding the kind of detail that I'd care about if I were evaluating that solution

digging into their GH repos surfaces https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-pu... which is at least more detailed if not quite a GMaps killer

qwertox• 3 years ago

I agree, if it weren't for your comment, I was about to ditch it as a candidate because it stopped zooming at a bad level. But your link shows that it contains a tremendous amount of data.

Edit: And I missed parents link to https://app.protomaps.com/downloads/small_map which is actually better.

bdon7• 3 years ago

I just updated https://protomaps.com it to use the more detailed file you found, thanks. So instead of a zoom level of 12 it should work down to 20+ - the actual dataset contains up to zoom 15, which is ~1 billion tiles in a single 110GB file.

It's important to me that I showcase the system working as advertised - you are free to download the 110GB file from that static storage with no restrictions. Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees. Before, it was on DigitalOcean Spaces, which is low latency but would cost me $1-2 out of my own pocket each time someone clicked the link.

The Lambda/Workers integration as described in the other comment is the preferred solution to this for developers operating this at planet scale. Of course, the static storage solution is perfect for small to medium-sized areas, even on GitHub Pages.

simmschi• 3 years ago

So one Lambda execution per tile?

wiredfool3• 3 years ago

It shouldn’t be required if the range requests are right and you have a compatible renderer in the client.

To expand a bit -- There are adapters for mapbox(gl/libre) and leaflet, and so on that will allow you to use a .pmtiles directly from the client with a range request supporting server, like s3 or nginx.

I've used protomaps basemaps to generate an Ireland and EU basemaps for a project from OSM data. I still haven't quite figured out the tuning for which features should be shown, combined, or excluded yet, and the generated layer names are sort of, but not quite compatible with similar base layers + styles from mapbox (which uses the same base data, but a different, proprietary conversion). This part takes a reasonable skill as a GIS person and some taste in the design of the feature symbology.

The downside of doing this on S3 directly is that you might wind up publishing a link to a 100 gig file that would cost $10 if someone just downloaded it. Lambda invocations seem cheap compared to that risk. OTOH, throwing it on a hetzner box is quite reasonable.

joshstrange4• 3 years ago

Like the other comment says I don't think it's quite that bad but also CloudFront (or whatever CDN you use) is going to cache the response. In my case I'm rendering restaurant locations on a map. I only have X number of items that each have a map of the restaurants they are available at so unless people pan/zoom around a lot I only expect a handful of lamba calls and the resulting tile data will be cached in CloudFront for everyone else.

bdon7• 3 years ago

Library author here - yes, one execution happens per tile, if you have CloudFront in front those will only happen when the PoP has a miss.

With 512MB RAM these generally complete in under 100ms so the unit costs of Lambda are within a few multiples of Cloudflare Workers (.5USD / million invocations)

Borg3• 3 years ago

Hmm, thats cool, but I need to have raster maps pushed to my endpoints. Do you know any easy way to use mvt maps and generate tiles from them on the fly (CGI backend + caching) and then use Leaflet to render those?

I have my own tiny personal project for maps renderer using Leaflet. For now I use 3 sources for tiles (Google, OSM and Esri) but having more sources would be very cool.

joshstrange4• 3 years ago

I don't personally know how to do that. It appears the PMTiles format does support raster maps but the dataset I use is only vector as far as I know.

divbzero• 3 years ago

Between this, Natural Earth, OpenStreetMap, USGS, and others, the availability of map data today would be stunning to early cartographers.

antod• 3 years ago

I tried my hand at pulling data from opentopology recently to use in blender and thought it was an amazing source of data for my country - I later realised that it seems like only the US and NZ are well represented there though (guess I was just lucky - thank you LINZ).

ChuckMcM• 3 years ago

Great stuff, at some point I feel like Google is going to have to try to extract even more revenue out of maps than they currently get and it will really empower these sorts of collaborative map systems.

I wish there was a way for people to fund satellite imagery that got pushed into these systems after purchase. Sunnyvale, for example, paid for a lot of imagery of the city that they use/used in staff discussions about traffic, zoning, etc. It would be nice if they could then push those images into the open data set.

Kadin• 3 years ago

There's really no technical reason why imagery like that couldn't be contributed, and IIRC Google does use municipally-owned imagery in its products. A single city might not be a big enough source for them to bother, I don't know, but I know particularly in Europe I have seen copyright notices for national and regional governments when looking at imagery.

Though there are a lot of sources for satellite imagery right now; Google may not be that hard-up for new stuff. I suspect the commercial vendors they work with probably image the entire CONUS area every 12 months or so.

The imagery that would seem to be more in-demand would be the aerial photos used at very high zoom levels. I'm not a geospatial expert but I think these images are combined with some sort of LIDAR or multi-spectrum imagery to have height maps in tandem with the visuals. That strikes me as pretty expensive to obtain.

jijji• 3 years ago

I like that they provide this data, however when you try to actually retrieve it, It looks like they went out of their way to make it as convoluted as possible to try to retrieve any of the actual data.... so you have to use DuckDB and then do the import using that? why not just support mysql dump files? why require someone to have DuckDB? is DuckDB that popular? also the links they provide don't work so it doesn't look like any of it's available.... how is somebody supposed to use this stuff? they require you to also have Amazon S3 with some query language that's non-standard to be able to talk to it.... I don't get it I'm sure I'm not the only one but it needs to be more generic instead of the way they're doing it

mdaniel5• 3 years ago

yeah, I thought about mentioning that it's kind of rude to lock that data behind AWS or Azure accounts, but realistically no one can be surprised that Big Cloud Providers want people to use more big cloud services

There's nothing stopping either AWS or Azure from granting `s3:Get` and `s3:List` on those buckets to enable unauthenticated reads, if they were thus interested

mikeocool• 3 years ago

I think they did?

aws --no-sign-request s3 ls s3://overturemaps-us-west-2/release/2023-07-26-alpha.0/

works fine for me.

dfgssdfgsdfg• 3 years ago

Is there any quicksand on the map? What about Lil Terry, is he on the map?

polemic• 3 years ago

It's unclear to me how Overture manages to "license wash" OSM data here.

Transportation: The OMF’s Transportation layer represents a worldwide road network derived from data in the OpenStreetMap project. This community-built data has been recast into the Overture data format which provides consistent segmentation of the data and a linear reference system to support additions of data such as speed limits or real-time traffic.

The OSM ODbL is crystal clear that OSM contributors have to be credited. I don't believe that CDLA Permissive v2.0 magically allows Overture to bypass it.

--EDIT: I missed that they're using different licenses per dataset, the transport theme is OBDL, which I'm sure will trip up users who are not careful.

uneekname5• 3 years ago

Overture could be seen as hostile to OpenStreetMap, but I do not think they are washing any licenses here. The article explicitly says that the CDLA license applies to data provided by Meta and Microsoft, not data from OSM. I think it's unfortunate that Meta and Microsoft aren't contributing their data to OSM instead of releasing it under a different license, but c'est la vie.

mikeyouse3• 3 years ago

Seems that any new data generated out of Overture Maps could be uploaded to OSM if they chose to do so? The CDLA license is permissive;

https://cdla.dev/permissive-2-0/

My read is that they're not going to avoid credited OSM, but rather they're going to credit OSM / maintain the license for the parts they use from OSM and then the rest will be CDLA 2.0 licensed for anyone to use.

kozikow• 3 years ago

Anyone loaded it to Bigquery yet?

I know this is "booo-google", but I just want to write some joins with other tables I happen to have in bigquery. I'm wondering if there is some "community" BQ rather than maintaining import of my own.

mike_d• 3 years ago

Administrative Boundaries: A global open dataset of national and regional administrative boundaries, this boundary data includes regional names which have been translated into over 40 different languages to support international use.

Oh that is going to be fun. If I recall correctly Google Maps alters the boundaries of places based on the views of the location the map is being requested from to avoid getting in the middle of disputes.

Not "correctly" showing boundaries is a crime in many countries.

Edit: here is a source https://qz.com/224821/see-how-borders-change-on-google-maps-...

risyachka• 3 years ago

Unless you are making a google maps rival, in 99% of cases you won't need those countries with controversial borders.

And unless you have millions and millions of users, no one will care even in those countries.

Mithridates1• 3 years ago

Does this add impedances to the OSM street segments?

zmix• 3 years ago

Will OSM incorporate this information into their own map?

anukul• 3 years ago

Has anyone been able to visually assess this dataset's accuracy against OSM / Google Maps etc for any given region?

Looks like it will be a while before that can be done, seeing as it uses a custom schema.

Overture Maps Foundation releases open map dataset

Related Stories