HN user

geodesk

16 karma

Developers of the fastest and smallest spatial database for OpenStreetMap data (free & open-source): www.geodesk.com; www.github.com/clarisma/geodesk

Posts0
Comments7
View on HN
No posts found.

The most apparent difference is storage consumption: A GeoDesk GOL file for a recent planet comes in at 80 GB -- as a pgSQL DB, it would require nearly 1 TB. GeoDesk spatial queries (especially intersects) leave pgSQL in the dust, even on low-end hardware (Throughput rates of 30 million features per second for bbox, 3M for 10K-vertex polygon, on a dual-core notebook with 8 GB RAM, SATA SSD; 4x this on a proper workstation). However, GeoDesk isn't intended as a general-purpose database. If you want to find the element with wikidata tag "Q1234", or all postal-code areas with ZIPs between 90000 and 92000, pgSQL will most likely perform better, assuming the data is properly indexed. One thing to note is that GOLs are read-only; future releases will allow incremental updates with fresh OSM data, but you cannot modify the data or add to it.

We designed GeoDesk with the assumption that users analyze/extract the OSM data with GeoDesk's tag-filtering/spatial operations, and then migrate the results to another database that matches their applications' needs. GOLs simplify distribution of OSM data. Internally, a GOL is organized into tiles (tens of thousands for a complete planet). With basic zip compression, these tiles are comparable in size to OSM-PBF (the most popular distribution format). As @pastage has already mentioned, converting OSM data into a GOL is significantly faster than importing into pgSQL (at least 20x). In the future, we envision that users will simply download GeoDesk tiles, skipping the conversion step altogether.

Another key difference is ease of development. GeoDesk queries return feature objects, obviating the need for object-relational mapping (For now, this is limited to Java and other JVM languages; we're considering a C++ port, which would pave the way for a Python API). GOLs replicate OSM object graphs, making it easy to explore the relationships between nodes, ways and relations: Find all residential streets within a city, count the number of speed bumps on each, find the crossroads, check which of these belong to bus routes, etc. All this can be done in pgSQL as well, but will require planning and proper table design to get it right.

This is essentially Spherical Mercator (EPSG:3857), except that instead of meters at the equator, planar distances are scaled so that the Earth's circumference equals 2^32 units (the maximum range of a 32-bit integer). This simplifies indexing, and also allows coordinate operations that cross the Antimeridian to wrap naturally. We chose this projection because it preserves angles and shapes at local scale, and it is already familiar to OpenStreetMap users. There are API methods to convert coordinates back to WGS-84 (degrees longitude/latitude) and obtain distances in meters (scaled based on latitude). For operations for which Mercator is not suitable (such as measuring the exact area of large landmasses), users can re-project the feature geometries using a library like Proj4J.

The underlying data of MapBox is 99.99% OSM (with some other datasets used for things like country borders). They do have a very nice vector-map viewer (previous versions were open-source, the newer releases have a more restrictive license that requires a subscription to their tile service).

If you just need map tiles, there are specialized providers (e.g. thunderforest.com) that have far more competitive pricing.

What specific data were you looking for? When did you last check out OSM? (There's been a huge push on data completeness in recent years that is now bearing fruit)

Combining datasets: You'll have to check the terms of the license agreements. In general, for ODbL-licensed data (like OSM), if you generate a "derivative database" (e.g. by merging multiple datasets), you will have to release this derivative database under ODbL terms as well. This would essentially rule out commercial data.

One way companies avoid this is by segregating the datasets, e.g. use a commercial dataset for hotel data, and use OSM data for streets and non-hotel POIs, without ever combining them into a single database. (But you'll have to check with an IP lawyer to be certain about the correct way to implement this).

OpenStreetMap is an excellent alternative. At this point, OSM data (which is freely available under the Open Database License) is equal (and often superior) to commercial offerings throughout much of the world. There's a great amount of detail, especially for tourism-related features (points of interest, difficulty level of hiking trails, opening hours of museums, etc.).

Google tends to have a greater number of commercial POIs (since it caters to potential advertisers), but OSM is catching up in this area as well.

Full disclosure: GeoDesk develops a spatial database engine for OpenStreetMap data, so obviously biased on this issue.

Indexing in GeoDesk GOL files is predominantly spatial (a quadtree-like structure combined with r-trees), with probabilistic indexing for the most frequent keys (`highway`, `building`, `amenity`, etc.). This means queries with a spatial dimension execute very quickly (bounding-box queries have a typical throughput of tens of millions of features per second on a simple dual-core notebook). GeoDesk is less suitable for non-spatial queries (e.g. "Where in the world is the Louvre?" may take several seconds on a planet-size database), but we'll consider enhanced indexing for these scenarios if there are enough real-world use cases to warrant it.

- Currently, GeoDesk is only available for Java (and other JVM languages). We're considering a future C++ implementation of the database engine (with possible Python bindings)

- We will be adding the capability to incrementally update GOLs via OpenStreetMap change files (.osc). We expect this feature to ship in Version 0.3 (slated for Q2 of 2023)