Translating OpenStreetMap data to HTML5 Canvas with Rust and WebAssembly

https://mary.codes/blog/programming/translating_openstreetmaps_to_HTML5_canvas_rust_wasm/

by todsacerdoti • 2 years ago

361 36 2 years ago

dakial1• 2 years ago

Author could use that queue time and some ML to forecast waiting times trends and create a real time Next-Best-Ride (NBR) model and plot those in the map. I'd find that useful the next time I'm pushing my 2 kids around that place...

maxerickson3• 2 years ago

The mentioned Overpass API uses sets as a sort of fundamental concept, so the code

    way[building][!name];foreach{(._;>;);out;}

can be replaced with

    way[building][!name];(._;>;);out;

The "(._;>;);" stanza is sort of dense and seems opaque without knowing what it's doing, but it's straightforward, returning the union "();" of the previous result "._;" and the children of the previous result ">;".

The change I propose above adjusts the ">;" to operate on all of the "way[building][!name];" results together, rather than each element one at the time.

enigmo• 2 years ago

this is a common enough need that

  out geom

does a similar operation in fewer characters

maxerickson3• 2 years ago

It doesn't do exactly the same thing, it inserts the lat/long into the parent instead of returning the nodes. Often doesn't matter, but it does if you are using a library that is expecting osm xml.

simpaticoder• 2 years ago

What a clear, step-by-step write-up of getting this POC (proof-of-concept) code running. She has picked as her audience experienced programmers, and one gets the sense that the write-up is as much for herself as for us. Which is great. We get to live vicariously through her and solve a small-ish (but non-trivial) problem with Rust, WASM and an OSM api with a cool visual payoff at the end. It's easy to let yourself not be impressed by straight-forward writing to a clear audience, because it looks so simple. But it's absolutely not simple and I'm glad to have found this blog and author. Thanks again, HN!

mxfh2• 2 years ago

The direct plotting from geographic coordinates (e.g., WGS84) can lead to visual distortions, especially for parks located further from the equator, where circles will appear flattened.

A practical solution would be to reproject the data to the Mercator projection or the appropriate local UTM Zone, ensuring accurate distances in cartesian units with no visible distortions of angles.

Alternatively, a quite straightforward method would involve scaling the longitudinal axis by the cosine of the latitude (cos φ [1]) at the center latitude of the area of interest. This adjustment provides a good approximation for small extents, like theme parks, without delving deeply into map projections.

[1] https://en.wikipedia.org/wiki/Longitude#Length_of_a_degree_o...

wodenokoto• 2 years ago

I thought all maps were drawn using web Mercator and that things were simply not large enough to be really distorted. Does gmaps and company do projected rendering?

sebstefan• 2 years ago

They do, if you zoom out they're a mercator that wraps around, but when they give you latitude and longitude instead of x and y, you're getting the real sphere stuff

samstave• 2 years ago

I used to know a lot more about rendering than I do today, but (forgive me if this sounds stupid) - I wonder if you could add to a gps coordinate area (say 1 sq. km. to also include that patche's NORMAL from the surface of the globe that you're projecting your map to, and have the 1km tile's image adjusted based on a orthographic view from that tile's NORMAL, so effectively - you're projecting each 1k square more directly from areal imagery?

Or is this already considered in modern projection methods?

modeless• 2 years ago

Google Maps on web uses perspective projection, unless WebGL is disabled in your browser. Zoom out and see!

lstodd• 2 years ago

I once experimented with something like this for OpenXcom is to first rotate the geographic coordinate system so that local equator/zero meridian intersection is at the center of the area of interest, then apply merkator or whatever other projection you like.

Otherwise it's a sea of hacks which doesn't get rid of latitude-induced distortions anyway.

After that being very annoyed with lat/long stuff I moved to N-vector representation.

https://en.wikipedia.org/wiki/N-vector

jillesvangurp• 2 years ago

Should not be an issue here as this is based on openstreetmap, which uses wgs84 coordinates everywhere. Changing the projection won't make that much of a difference for small features like parks or buildings.

You are right that this is an issue if you start plotting e.g. circles using a naive algorithm as the x and y scales in degrees are simply not the same in meters. Which is an issue if you want to draw circles and rectangles on a map.

The solution to that is to do it in meters and then translate the points from some origin using e.g. the Haversine distance. A related issue you have there is that distance algorithms aren't accurate either but it's close enough as an approximation over small distances.

I have some algorithms for that in my jillesvangurp/geogeometry library. I also have some UTM coordinate conversion algorithms in there that I recently added.

mxfh2• 2 years ago

The distortion is still there, even for small features as seen in the OP examples of circular features of the park. This due to the changing length of longitudinal unit at a given latitude.

Historically this is the reason web mapping picked up Mercator in the first place, since it preserves angles and relatives sizes (conformal) at city scales, which is the primary use case of those.

WGS84 is a CRS first, not meant to be used as cartographic projection for a known local scope. (for some Plate carée is more of a plot than a projection)

I’m not talking about converting between local CRS for surveying an local plate consistency. You just want the display to look not visibly distorted in a way that’s not intentional. Florida is still quite forgiving here. Eurodisney not so much.

https://commons.wikimedia.org/wiki/File:Plate_Carr%C3%A9e_wi...

naitgacem• 2 years ago

I have tried to use Canvas on Android to draw some custom view (chart). I still have a PTSD from that experience. I have an exceptionally great amount of respect for people who can do this sort of stuff.

liftm• 2 years ago

Well, if you're playing with wasm like this post, you could sidestep canvas and render the image on the wasm side. c.f. https://github.com/plotters-rs/plotters-wasm-demo

ativzzz• 2 years ago

I did something similar but used SVG instead - it didn't feel that bad

beeboobaa• 2 years ago

While I haven't performed any benchmarks about whether Rust or JavaScript processing is faster, I decided to use Rust and WebAssembly because there will be a lot of coordinates to process, and Rust generally performs faster at large amounts of data processing.

Is this really true for Rust compiled to WebAssembly in the browser?

davidmurdoch• 2 years ago

Depends on the data types and if the WASM code can make use of SIMD.

grovesNL• 2 years ago

It depends on the use case (e.g., how much you're interacting with web APIs) but I'd generally expect this to be the case for data processing at least.

During compilation from Rust to WebAssembly you have the opportunity to apply ahead-of-time optimizations that would be too expensive or impossible to apply to JavaScript during execution, so this can improve performance pretty significantly over JavaScript.

afavour• 2 years ago

It’s an interesting question. When it comes to processing large amounts of data I’d actually be more concerned about JavaScript’s memory usage than strict processing performance. Certainly you could use JS smartly with typed arrays and such to minimise memory impact but you'd basically be manually creating a lot of the convenience that Rust gives you out of the box.

Solvency• 2 years ago

If they have a public API why did she have to crop and manually export stuff? What is she exporting, data? If so what is the API doing?

Ajedi32• 2 years ago

Looks like she wasn't actually exporting anything, just using the export feature as a convenient way to get the coordinates of the bounding box she wanted to search in.

gerogerke• 2 years ago

For anyone interested, I have a similar blog-post [0], in which I explore rendering OSM data using WebGL.

[0] https://gero.dev/blog/webgl-render-osm-streets

_neil• 2 years ago

Nice article. It's good to see regl is still being used.

wvh• 2 years ago

This is a cool project; it has just enough meat on its bones to learn a bit about WASM and mapping data, and an attainable goal with a clear, visual result after some effort. I just spent half an hour to try to get this to work myself to see how far WASM has come. Not sure I'll be doing something with this specifically, but good to keep in the back of my head for future challenges along the same vein.

butz• 2 years ago

Here is an idea: use Mapsforge binary format for map data. Map sizes are quite small, especially useful for not to huge areas. Implementing HTTP Range requests might be an interesting exercise in reducing bandwidth usage.

coder543• 2 years ago

An existing solution:

PMTiles readers use HTTP Range Requests to fetch only the relevant tile or metadata inside a PMTiles archive on-demand.

https://docs.protomaps.com/pmtiles/

GeorgeHoneywood• 2 years ago

Somewhat bizarrely I ended up doing exactly that as a uni project. Writing a parser for the Mapsforge format in TypeScript was a bit of a weird experience. The funkiest bit was trying to get the range request stuff to work properly offline via service workers.

My only regret is not putting more effort into getting WebGL working, I just used the plain canvas and it was a bit of a bottleneck.

demo: https://files.george.honeywood.org.uk/final-deliverable/#16/...

blog: https://george.honeywood.org.uk/blog/map-from-the-ground-up/

emporas• 2 years ago

I was looking at that project [1] trying to understand what this article explains in a more simple way. There is some non-trivial complexity of how to put together wasm-pack, web-sys, js-sys and wasm-bindgen functions to work well together with javascript for a web page or a browser extension. Nice article!

[1] https://github.com/drakerossman/hackernews-userscript

NoPedantsThanks2• 2 years ago

Wasn't there a post on here recently about OSM going to vector-based tiles? I did a quick search and it seems like this is already a widely-offered option. Can anyone jog our memory on what the change is?

maxerickson3• 2 years ago

OSM foundation is going to host vector tiles and build a stack to keep them up to date with ongoing editing.

You are probably talking about https://news.ycombinator.com/item?id=39339182

mikeocool• 2 years ago

The OSM Foundation is building out a stack to generate 'minutely' vector tiles -- meaning the if you edit the map, it'll be reflected in the vector tiles within a few minutes.

Most vector tile sets today generated from OSM are hours behind the current state of the map data in the best case, much more commonly they are months behind.

NoPedantsThanks2• 2 years ago

Thanks for the replies. Sounds like a nice addition.

HumblyTossed• 2 years ago

In case you're hungry after reading this, she has a post on how to make creamy grits:

https://mary.codes/blog/recipes/the_secret_to_the_creamiest_...

I love grits, but have never tried baking soda (or even heard of it). Gonna have to try this.

londons_explore• 2 years ago

So the data seems to come from the overpass API as XML, gets a bit of transformation in rust, passed back to javascript for drawing...

To me, that seems like the wrong design for performance... Drawing you probably want to use webgl shaders for so you can do zooming and panning at 60 fps...

And downloading the data as XML seems like a bad idea if you're drawing city-sized areas because the XML will be huge and contain plenty more detail than you need. Instead you need a server-side component to pack simplified and quantized polygon data for the necessary scale into a binary datastructure for sending to the client.

And thats exactly how bing/google maps/apple maps work...

Translating OpenStreetMap data to HTML5 Canvas with Rust and WebAssembly

Related Stories