HN user

nivekney

47 karma
Posts1
Comments18
View on HN

Map-reduce as a pattern might be on its way back. Hear me out. High localization wins even when coverage is not super great -- just map shards of the corpus and reduce the learnings. Rinse and repeat, do as many rounds of map and reduce to traverse the corpus until converge. This can also work well when the cluster is combined with different agents, they are tasks equally by prompts anyway.

It's based on ParlayLib, which is for shared-memory multicore machines. Highly suspect that they moved the algorithms on to distributed systems.