Is anybody else loading their database by tailing log files?
https://news.ycombinator.com/item?id=842122I recently had a brief chat with a couple of developers working on different data-heavy websites, who were both using an interesting pattern for filling their databases.
Their data-gathering components (pulling from external sources like crawlers and APIs) would append new data to the bottom of a log file.
Another process sat doing something like a 'tail -f' on the same file, and parsed and added the updates to the database.
This seems like it might solve some problems for my case:
- Very easy to recreate the database if the schema changes or things blow up, just reread the log files
- Good history for debugging
What worries me is that it feels funky using files for IPC, and I can't find any examples of this being used elsewhere.
So, is anyone else using this pattern, or have any references to it that I'm missing?