Ask HN: What is the best way to calculate percentile of streaming data? 7 years ago
No I am not using Kafka. It is just basic python server. I want to calculate what is the nth percentile of size of my incoming request object over past one hour.
I don't want to store size of each request in memory. It will eat so much of my RAM.
Incoming traffic:
- 1st batch
--> 60 requests
---> size of 1st request is 10kb
---> size of 2nd request is 2kb
...
...
- 2nd batch
--> 10 requests
---> size of 1st request is 5kb
---> size of 2nd request is 8kb
...
- 100th batch
I am talking about percentile(10th, 50th, 95th) size of request.