Is anybody else loading their database by tailing log files?

https://news.ycombinator.com/item?id=842122
by petewarden • 17 years ago
5 6 17 years ago

I recently had a brief chat with a couple of developers working on different data-heavy websites, who were both using an interesting pattern for filling their databases.

Their data-gathering components (pulling from external sources like crawlers and APIs) would append new data to the bottom of a log file.

Another process sat doing something like a 'tail -f' on the same file, and parsed and added the updates to the database.

This seems like it might solve some problems for my case:

- Very easy to recreate the database if the schema changes or things blow up, just reread the log files

- Good history for debugging

What worries me is that it feels funky using files for IPC, and I can't find any examples of this being used elsewhere.

So, is anyone else using this pattern, or have any references to it that I'm missing?

Related Stories

Loading related stories...

Source preview

news.ycombinator.com