HN user
The devil is in the details probably, but if you have a single datacenter and all writes are coming from thousands of machines ("edge"), yes, it may make more sense to set up a single LogDevice / Kafka cluster and have all the edge hosts write to it directly.
That's a good question. There are multiple reasons for this. I'll briefly mention two of them. One is the high fan in ratio - millions of machines are writing relatively small blobs of data, so the middle layer serves as an aggregator (which saves backend's IOPs, number of connections, etc). Another reason is the volume of metadata - it would be inefficient to keep all the LogDevice-level metadata on each of the producer hosts.