Ask HN: Why isn't a content addressable cache built into the Internet?
https://news.ycombinator.com/item?id=2277607I have a simple internet protocol in mind. Does it already exist? If not, why not?
The idea is that there should be a content-addressable request protocol. By presenting a (20 byte) hash the server responds with a chunk of data (say, 2M in size) which is associate with that hash. If a publisher (NetFlix) were to offer their content using this protocol, they would transmit to each authorized viewer, a list of 500 hashes for a 1 GB movie (total size of hash list - 10K). The client then requests each chunk by hash key in order to stream a movie.
Any ISP (Comcast, Level 3) seeing this protocol coming over their network, realizes that they can service a chunk request by returning the cached data, instead of forwarding it upstream to the source to fetch it over and over again. Routers could incorporate the protocol into their stack, so they can short-circuit high volume chunk requests out of a local cache (from several gigabytes to terrabytes). Once an ISP has the chunk data, they can serve it 1,000 times without requesting it from the origin server. They can manage their cache purely based on demand for each chunk.
With this protocol widely in place, I think it would be possible for 100M households to simultaneously watch a movie when it becomes available for download. I don't think that's possible using the current Internet protocols and CDN's.
Would this not be far superior for the network to transparently handle caching this way, rather than the messy system of CDN's we have today?
Does copyright law or the DMCA make it impossible to operate a system like this?