Ask HN: 3k museums submitting data. How?
https://news.ycombinator.com/item?id=30887223I'm helping with a project that will see ~3k small museums submitting their collections data to a central data lake. It can be assumed that most museums struggle with technology and have little / no resource in-house. The data itself varies from an xls / CSV file to a database dump, to all flavours of thing in-between. File size: almost nothing all the way up to maybe 100mb. The files will change regularly, so the solution will need to cater for this.
My questions are twofold:
First, what options do I have on the data collection side? Should we hook into something familiar like Dropbox or are there other seamless / easy to manage ways of requesting a local data sync? Making a form on a submission website was our first thought but data transfer is likely to be slow / impossible for some museums so some kind of desktop file-watcher seems less brittle, maybe?
Second: I know I haven't given nearly enough information for detail, but are there existing technologies on the receiving end that could ingest disparate, non-mapped, schema-less data of all flavours? Should we look at something like Node Red or Parabola do so some mapping, or are there other generic data receiving technologies we should be aware of that can cope with a multitude of data types, shapes, sizes?
I realise this isn't enough information for a complete answer but even some links / hints / ideas of what to look out for would be appreciated. Thanks, HN :-)