HN user
We speak about this is much more detail in this post (https://0x65.dev/blog/2019-12-05/a-new-search-engine.html), but in short, we prototyped our search initially with data we purchased from data-brokers. Once the concept was proven and HumanWeb was deployed (2015/2016), we rely only on our data.
[Disclaimer: work at Cliqz] We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here :
How we collect data : https://www.0x65.dev/blog/2019-12-03/human-web-collecting-da...
How we build the search using this data: https://www.0x65.dev/blog/2019-12-06/building-a-search-engin...
Feel free to peruse these posts and ask questions!
Hi! Yes, we have played around with character and trigram level neural network language models. Also, we experimented with training a supervised neural network based on a misspellings dataset for the corrector.
Unfortunately, we had trouble getting the performance to a point where they could replace this system. It is definitely something we will revisit soon though!
Sure. cmputer gets corrected, not omputer.
Just to provide some additional information, our library Keyvi(https://github.com/KeyviDev/Keyvi) has a very fast implementation of an FST based spell correct.
[Disclaimer: work at Cliqz] Both techniques have their share of upsides and downsides, infact we also use an FST based model to perform splits i.e donaldtrump ---> donald trump. The problem with the FST approach is when the prefix has a typo. I tried the spell correct on the website you linked to, it seems to fail when the first character is incorrect. Thanks for the information though, it is always interesting to read and learn from other approaches :)
[Disclaimer: I work at Cliqz]
We will have a blog post tomorrow on this very topic, but in short, we use a combination of Keyvi, Granne (both in-house) along with Cassandra and RocksDB.
Though our approach mentioned in this blogpost significantly reduces the storage needed to host the index, we still have an index of around 50 TB of data.