marklit

Thanks for pointing that out. I'll update the post.

I love that idea. I don't have time for anything elaborate today but I dropped two visualisations at the bottom of the post.

141 scenes in this dataset have MP4 counterparts.

I can't get the vector basemap to render in QGIS properly with any other projection. I love using EPSG:3301 for Europe, etc.. but loads of strange things happen. Even 4326 is an issue.

Glad you enjoyed it.

There should be enough SQL in the blog to re-purpose extracting out the Wildberries locations and seeing where they land on top of. I've never heard of this firm before you mentioned it.

From Google:

Citibank operates over 2,300 ATMs within more than 600 U.S. branches, with a total network of over 65,000 fee-free ATMs

So the 57,163 Citibank locations are probably a combination of their branches and ATMs.

Update: I reviewed Alltheplaces a while back, they scrape company websites for store locations. They reported 68,227 locations for Wildberries. ATP is one of the sources Overture use but they seem to use 1.55M of the records from their 19M-record dataset. https://tech.marksblogg.com/alltheplaces.html

I'm having a hard time tracking down the OSM-specific actions around this but Australia moved 1.8M back in 2017 due to continental shifting https://www.ga.gov.au/news/news-archive/australia-officially...

The Airport in Tartu, Estonia had a navigation upgrade last year in order to help mitigate navigation jamming that's taking place in the region. https://www.eans.ee/en/uudised/tanasest-saab-tartu-lennuvalj...

Defcon had a great talk on all the different navigational systems pilots can use and a note at the end that these shouldn't be decommissioned at the rate they're experiencing atm https://www.youtube.com/watch?v=wSVdfOn737o

From what I've seen in news reports, China has built a lot of tower blocks that are 10s of floors, rather than the 4-5-floor buildings I saw when I worked in and travelled around India.

India has ~4x the population of the US so the ratio of buildings isn't much of a surprise.

The US has only used 25% of its land. Overture published a land use dataset a while back that could go some way to verify how much land on earth is urban, covered in forest, etc.. That might only need a single SQL statement given how they structured their data.

I have some analysis around that topic in the middle of this post: https://tech.marksblogg.com/overture-land-cover.html

That 5 TB of data will probably be 3-400 GB in Parquet. Try and denormalise the data into a few datasets or just one dataset if you can.

DuckDB querying the data should be able to return results in milliseconds if the smaller columns are being used a better if the row-group stats can be used to answer queries.

You can host those Parquet files on a local disk or S3. A local disk might be cheaper if this is exposed to the outside world as well as giving you a price ceiling on hosting.

If you have a Parquet file with billions of records and row-groups measuring into the thousands then hosting on something like Cloudflare where there is a per-request charge could get a bit expensive if this is a popular dataset. At a minimum, DuckDB will look at the stats for each row-group for any column involved with a query. It might be cheaper just to pay for 400 GB of storage with your hosting provider.

There is a project to convert OSM to Parquet every week and we had to look into some of those issues https://github.com/osmus/layercake/issues/22

I'm hoping to use most of this system for the next 10 years. At some point, I want to add some beefy GPUs to it when I get back into 3D again.

A good percentage of Pakistan did this recently and removed 35% of demand off of their grid https://www.ted.com/talks/jenny_chase_solar_energy_is_even_c...

There are ~15 GB of SAR imagery at the bottom being rendered as is from GeoTIFF files. On my 2020 MBP rendering that amount of data in QGIS would lag without building mosaics and tiles.

The Parquet pattern I'm promoting makes working across a wide variety of datasets much easier. Not every dataset is huge but being in Parquet makes it much easier to analyse across a wide variety of tooling.

In the web world, you might only have a handful of datasets that your systems produce so you can pick the format and schemes ahead of time. In the GIS world, you are forever sourcing new datasets from strangers. There are 80+ vector GIS formats supported in GDAL. Getting more people to publish to Parquet first removes a lot of ETL tasks for everyone else down the line.

70% of daytime RGB sat imagery is covered by clouds. I'm not sure how easy it would be to spot if clouds were covering a city's lights at night.

I've only seen Maxar publish one night time image and that was of Dubai. I suspect smaller buildings in not so well lit areas could end up getting missed out.

SAR imagery would work well for seeing at night and through clouds but I'm not sure what the state of AI building footprint detection is with SAR atm.

I haven't done any deep dives into their road data but there was ~80 GB of it, mostly from TomTom, in the August release. I think the big question would be how much overlap there is with HERE and how would the metadata compare.

TomTom did a few write ups on their contribution, this one is from 2023: https://www.tomtom.com/newsroom/behind-the-map/how-tomtom-ma...

If you have QGIS running, I did a walkthrough using the GeoParquet Downloader Plugin with the 2.75B Building dataset TUM released a few weeks ago. It can take any bounding box you have your workspace centred on and download the latest transport layers for Overture. No need for a custom URL as its one of the default data sources the plugin ships with. https://tech.marksblogg.com/building-footprints-gba.html

Amazon, Esri, Grab, Hyundai, Meta, Microsoft, Precisely, Tripadvisor and TomTom, along with 10s of other businesses got together and offer OSM data in Parquet on S3 free of charge. You can query it surgically and run analytics on it needing only MBs of bandwidth on what is a multi-TB dataset at this point. https://tech.marksblogg.com/overture-dec-2024-update.html

If you're using ArcGIS Pro, use this plugin: https://tech.marksblogg.com/overture-maps-esri-arcgis-pro.ht...

primary way to get a clean-ish OSM shapefile

Shapefiles shouldn't be what you're after, Parquet can almost always do a better job unless you need to either edit something or use really advanced geometry not yet supported in Parquet.

Also, this is your best source for bulk OSM data: https://tech.marksblogg.com/overture-dec-2024-update.html

If you're using ArcGIS Pro, use this plugin: https://tech.marksblogg.com/overture-maps-esri-arcgis-pro.ht...

I suspect web apps that "query" the GPKG files. Parquet can be queried surgically, I'm not sure if there is a way to do the same with GPKG.

The report doesn't appear to be public.

Fixed those. Cheers for pointing them out.

From discussions I've seen online, a lot can fail on satellites so there is a bias to do as little as possible on them. There does appear to be plenty of bandwidth to transmit back to ground stations.

The real win will be satellite-to-satellite transmissions where any data collected by the constellation is passed to the satellite that'll next fly over a ground station. This will lower the time from capture to analysis considerably. The fresher the data, the more valuable it is.

I've got other open data feed posts on the site for Capella, ICEYE, Umbra and Maxar.

To anyone considering going to all this effort, consider doing this work on OpenStreetMap. 50K contributors make OSM a bit better every month but a good map is never finished. https://rapideditor.org/edit

Fixed. Thanks for spotting that.

Sorry, I just added a link to a live demo in the post: https://pnorman.github.io/tilekiln-shortbread-demo/

OSM doesn't have the same metadata as this project.

Thanks for clearing that up.

The JSON files in the S3 bucket can be downloaded in isolation from any other format and they're pretty small. I did some geo-enrichment of them in a previous blog post where I found the country and waters of Umbra's ship imagery. https://tech.marksblogg.com/yolo-umbra-sar-satellites-ship-d...

The above relies on network calls to OSM but with Capella, I found a way to get the smallest GeoFabrik partition for each image using a single GeoJSON file. The code could probably be modified to work on Umbra's feed with a bit of work. https://tech.marksblogg.com/capella-open-data-free-satellite...

Lastly, Umbra have ~24 locations they image frequently but there is an 'ad hoc' folder with a lot of subject names in English that give away the location and subject matter. This might be easier to look through for interesting imagery.

aws s3 --no-sign-request ls 's3://umbra-open-data-catalog/sar-data/tasks/ad hoc/' | grep -i 'tesla\|nvidia\|saudi'

DuckDB's file format is OLAP-based but supports CRUD operations. I've created DBs with it with 10s of millions of records and gotten amazing query speeds.

Efficient querying using indices over S3 for their format isn't something I've looked into just yet.

The GPKG files https://github.com/marklit/osm_split produces are SQLite files under the hood and come with an R-Tree Spatial index so they'll load into QGIS and other GIS software quickly.

Green Idea | Estonia | REMOTE | Full-time | Windows + DirectX C++ Developer

I'm looking for a C++ developer to build Geospatial / GIS visualisation and editing tools for Windows. These will work in both 2D and 3D. These will act more like desktop applications like AutoCAD and 3D Studio Max rather than full-screen video game applications.

The project is likely to be iterative so if you thrive on agency and being a part of the brain-storming process, you should feel at home.

Knowledge of the following is expected:

* C++17

* Computer Vision + Graphics, photogrammetry

* CUDA

* DirectX

* DuckDB and SQLite3

* GDAL and GEOS

* Packaging Windows Applications for distribution

* Python

* QGIS & ArcGIS Pro

* WinUI 3 and Windows Ribbon

We're based in Estonia but flexible with time zones between here and California.

Contact me via https://www.linkedin.com/in/marklitwintschik/ to apply for the role.