kylebarron

See recent discussion in [0]

[0] https://news.ycombinator.com/item?id=47482185

Looks great!

Another seemingly extremely similar project released in the last few days: https://github.com/raulcd/datanomy

https://github.com/manzt/juv

Agreed, I find this to be a super productive environment, because you get all of vscode's IDE plus the niceties of Jupyter and IPython.

I wrote a small vscode extension that builds upon this to automatically infer code blocks via indentation, so that you don't have to select them manually: [0]

[0] https://github.com/kylebarron/vscode-jupyter-python

Equal distances to each adjacent neighbor: https://www.uber.com/blog/h3/

This worked for me last month on a Mac M2: https://www.portingkit.com/game/278

https://visgl.github.io/react-map-gl/ is a react interface to MapLibre

I develop Lonboard [0], a Python library for plotting large geospatial data. If you have small data (~max 30,000 coordinates), leaflet-based Python libraries like folium and ipyleaflet can be fine, but because Lonboard uses deck.gl for GPU-accelerated rendering, it's 30-50x faster than leaflet for large datasets [1].

[0] https://developmentseed.org/lonboard/latest/

[1] https://developmentseed.org/lonboard/latest/how-it-works/

It can read from HTTP urls, but you'd need to manage signing the URLs yourself. On the writing side, it currently writes to an ArrayBuffer, which then you could upload to a server or save on the user's machine.

Arrow JS is just ArrayBuffers underneath. You do want to amortize some operations to avoid unnecessary conversions. I.e. Arrow JS stores strings as UTF-8, but native JS strings are UTF-16 I believe.

Arrow is especially powerful across the WASM <--> JS boundary! In fact, I wrote a library to interpret Arrow from Wasm memory into JS without any copies [0]. (Motivating blog post [1])

[0] https://github.com/kylebarron/arrow-js-ffi

[1] https://observablehq.com/@kylebarron/zero-copy-apache-arrow-...

That's why I'm working on the GeoParquet spec [0]! It gives you both compression-by-default and super fast reads and writes! So it's usually as small as gzipped CSV, if not smaller, while being faster to read and write than GeoPackage.

Try using `GeoDataFrame.to_parquet` and `GeoPandas.read_parquet`

[0] https://github.com/opengeospatial/geoparquet

Your issue is that you're using the default (old) binding to GDAL, based on Fiona [0].

You need to use pyogrio [1], its vectorized counterpart, instead. Make sure you use `engine="pyogrio"` when calling `to_file` [2]. Fiona does a loop in Python, while pyogrio is exclusively compiled. So pyogrio is usually about 10-15x faster than fiona. Soon, in pyogrio version 0.8, it will be another ~2-4x faster than pyogrio is now [3].

[0] https://github.com/Toblerity/Fiona

[1] https://github.com/geopandas/pyogrio

[2] https://geopandas.org/en/stable/docs/reference/api/geopandas...

[3] https://github.com/geopandas/pyogrio/pull/346

Sorry, this is not true _at all_ for geospatial data.

A quick benchmark [0] shows that saving to GeoPackage, FlatGeobuf, and GeoParquet are roughly 10x faster than saving to CSV. Additionally, the CSV is much larger than any other format.

[0] https://gist.github.com/kylebarron/f632bbf95dbb81c571e4e64cd...

Is this open? Is there a way to read more about it? Issues/pull requests?

nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box.

As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)

pmtiles are only the counterpart to the "overviews" part of COG. pmtiles doesn't give you analysis-ready data. For that, look to FlatGeobuf or GeoParquet (once it gets a spatial index in v1.1 of the spec)

10,000 coordinates are certainly not enough to see the difference, but at some scale this would be faster on the GPU.

This is implemented in an nvidia geospatial library call cuspatial: https://docs.rapids.ai/api/cuspatial/legacy/api_docs/spatial...

I'm working on bringing GeoRust algorithms [0] (which include boolean operations) to the web via WebAssembly. See this blog post [1] for an intro. There's also separate work on binding GEOS to Wasm [2], but I'm more excited about GeoArrow in the long term because you can interpret Wasm geometries in JS without any copies across the boundary [3].

[0] https://docs.rs/geo/latest/geo/#algorithms

[1] https://observablehq.com/@kylebarron/prototyping-georust-geo...

[2] https://github.com/chrispahm/geos-wasm

[3] https://observablehq.com/@kylebarron/zero-copy-apache-arrow-...

I'm surprised this link works and doesn't redirect given https://macwright.com/2022/09/15/hacker-news.html

The line visuals at the bottom are not using Mapbox for the data rendering. Rather they're using the open source Kepler.gl [0], (a user-friendly wrapping of the deck.gl library [1]). These can use Mapbox for the underlying basemap, but the data rendering is done separately.

(This is easy to tell if you look at the page source. The map at the bottom is an embed from a static HTML kepler.gl map [2]). Disclaimer: I've contributed to deck.gl

[0] https://kepler.gl/

[1] https://deck.gl/

[2] https://senseable.mit.edu/bike-trafficking/assets/kepler.htm...

DuckDB is not backed by Arrow, but by something similar enough that it can be zero copy to Arrow in many cases https://duckdb.org/2021/12/03/duck-arrow.html

Most of these examples go up to zoom 10 or so because they cover a wide area. It's not prohibitive to increase that zoom in a small area because the total number of tiles will still be low. Also keep in mind that for vector data it's possible to "overzoom" and allow the client zoom levels to be higher than the physical data levels.

In terms of self hosted maps for outdoors use, I have some open source repos from a few years ago that may help with that. [0] is a style based on the openmaptiles vector tile schema for outdoors purposes. I believe that readme has some overall instructions, plus other repos in the org are for making data. (Nowadays planetiler is probably the best way to make OSM vector tiles though)

[0] https://github.com/nst-guide/osm-liberty-topo

Maybe due to SIMD?

pandas added a copy-on-write API in 2.0, so under the hood the Arrow columns are still effectively immutable.

https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#copy...

I didn't think Fatmap was based on mapbox at all? I thought it was their own custom renderer

On the contrary, it can use and interface with numpy quite easily: https://github.com/PyO3/rust-numpy

I think you're referring to the "inlay hint" API, which is different from being able to visualize any HTML in the current text editor, like Hydrogen is able to: https://cloud.githubusercontent.com/assets/13285808/20360886...

But with Hydrogen you could do that in a regular Python script without needing to create a notebook or think in terms of cells. There's an example of this in the Hydrogen readme: https://github.com/nteract/hydrogen#hydrogen-

That's exactly my complaint though. I don't want to work in a notebook frame of mind. I don't want to have to demarcate `#%%` every time I run code (especially because you'd need to scroll down to put another `#%%` where the code you're running should stop.)

As I noted in a sibling comment, I wrote a quick extension [0] that makes this easier by automatically inferring code blocks, but VSCode Jupyter is lacking in other ways: you can't have multiple Jupyter kernels running at the same time and it doesn't show you runtime completions in the editor (necessary for compiled packages without type support).

[0] https://github.com/kylebarron/vscode-jupyter-python

I completely agree. For the last couple years I've used VSCode for everything _except_ interactive Python development because Atom + Hydrogen is just too good.

I'm giving VS Code + its Jupyter extension a shot for now. It's significantly worse than Hydrogen, but I tried to make it a little less bad with a quick extension to auto-infer code blocks [0].

[0] https://github.com/kylebarron/vscode-jupyter-python