Do large websites like Twitter and Facebook log every single POST / GET request?
https://news.ycombinator.com/item?id=42821873I’m relatively inexperienced in handling large-scale applications, but I’ve recently been tasked with scaling a web application for a large city. One major challenge I’m facing is logging HTTP requests efficiently.
Currently, we use Kibana and keep logs for 4 weeks, and we’re handling around 1 million logs per day. As more service are added and traffic increases, I’m concerned about storage costs, performance impact, and best practices for managing logs at scale. Most of the devs working here are also fresh out of school with limited experience.
Given that platforms like Twitter, Facebook, and Reddit handle millions to billions of requests per day, do they log every single GET and POST request, or do they use sampling, aggregation, or filtering?
What are the trade-offs between complete logging vs. selective logging, especially in terms of performance, storage, compliance, and debugging?
If anyone has experience working with large-scale logging systems, I’d love to hear how different companies approach this problem. Any guidance would be greatly appreciated!