Ask HN: Interested in a centralized large-scale crawl architecture?

I have been using Nutch for a while and find it quite low-level. The batch based indexing system is good but doesn't work well in many scenarios.

I am talking about a single point, auto-updated crawl database. The distributed database will support APIs to inject/remove URLs, fetch content for a URL and register callbacks when a crawl is over, etc.