Local development and huge databases?

https://news.ycombinator.com/item?id=7353104
by mdomans • 12 years ago
1 1 12 years ago

Question for guys working on any data analytics projects - local development and working on features when the database we touch is huge. For some time now I have the problem of working with huge datasets in relational databases where having a local copy of such DB is simply not possible. I have a way of working around the problem by utilising either a mix of local editor and remote iPython or having some mocked data. That of course causes some problems with performance, since you can quite easily write slow code. Any ideas?

TL;DR: How to develop code when you need a huge database for work (more than 40GB)?

Related Stories

Loading related stories...

Source preview

news.ycombinator.com