Riot – Full-text search engine in Go

by veni0 • 9 years ago

221 66 9 years ago

dest• 9 years ago

A comparison with Yacy would be interesting IMHO

dpcx3• 9 years ago

Not that I'm against people building tools in their language of choice, but how does this compare to Sphinx (http://sphinxsearch.com/)?

didip• 9 years ago

Last I used Sphinx, it is tied to MySQL. Is that still true?

If so, then Sphinx could be a deal breaker to some.

dpcx3• 9 years ago

Sphinx can connect to MySQL or Postgres, but can also read XML and [TC]SV, among others. Does this do something better? The examples seem to be where the user is providing the data to the indexer, which Sphinx can also do.

mrweasel• 9 years ago

It hasn't been tied to MySQL in the last 10 years, so I'm wondering if it ever was.

You can connect to Sphinx using a MySQL client, use it as a MySQL storage engine or using MySQL as a data source. But it's not specifically tied to MySQL.

pQd2• 9 years ago

It's worth mentioning that the original Sphinxsearch project has been stagnant for the past year. There's a new lively fork - https://manticoresearch.com/

Beyond being a user of both I don't have affiliations with either of them.

dpcx3• 9 years ago

Good to know. Any idea what happened?

Jemaclus2• 9 years ago

Also, for consideration: Bleve (https://github.com/blevesearch/bleve)

I'm in the process of building my own search engine (as a learning exercise, but also because it's related to my day job). I've learned that it's one thing to write a full-text search engine, like this one, and it's quite another to do field-specific searches with faceting support and so on, like Algolia and Lucene-based search engines do.

That said, this is clean and simple. I like it. I can definitely learn from this.

CSDude• 9 years ago

Last time I checked, bleeve was very slow compared to Lucene, I really hope it gets better over time.

markpapadakis4• 9 years ago

Supporting faced search and other functionality requiring access to per document field-values is just an extension over the core IR functionality.

Tracking (document, field) values can be used for query by range or by geolocation primitives (that's what Lucene does, where it will index that data into a special tree-like structure, and for each query, it will build a custom 'iterator' and use it along with other iterators to match documents), and for static ranking of matched documents.

BTW, Lucene and Algolia are vastly different in terms of the underlying architecture.

lobster_johnson• 9 years ago

What is Algolia's underlying architecture like? Are there any papers or code?

Jemaclus2• 9 years ago

That's why I mentioned them separately. :)

StreamBright• 9 years ago

I was planning to write one too, wanna team up? :)

amelius• 9 years ago

Could you please include a bibliography in your project? That way, others can more easily find which techniques you are using, and of course it can help others (or even your future self) in figuring out how the code works.

unicornporn• 9 years ago

Kind of taken https://about.riot.im/

lylejohnson• 9 years ago

Not to mention http://riotjs.com/.

speps• 9 years ago

What about https://www.riotgames.com ? More so because they have an engineering blog, which is very interesting : https://engineering.riotgames.com/

krautsourced• 9 years ago

Also http://luci.criosweb.ro/riot/

tenken• 9 years ago

Also for consideration: https://github.com/coleifer/scout

RESTful search server written in Python, powered by SQLite.

wolfgarbe• 9 years ago

It seems the whole index is kept in RAM. Thus the index size is limited by the amount of RAM available. This explains the impressive indexing and search performance (1M blog 500M data 28 seconds index finished, 1.65 ms search response time, 19K search QPS) The Persistent storage data is stored to the hard disk solely when the program closes. The data is then restored from the hard disk when the program restarts ( https://github.com/go-ego/riot/blob/master/docs/zh/persisten... ). This is a limited approach compared to Lucene/Solr/Elasticsearch LSM which handle high-volume inserts to its indexes with a log-structured merge-tree (LSM) and where the index size is only limited by the available hard disk space.

markpapadakis4• 9 years ago

1.65ms for what kind of queries? Also, is that 1M blog posts, all weighting 500mb in total size(characters)?

ethanwillis2• 9 years ago

I wonder if they're using succint data structures.

I'm in bioinformatics and the first time I implemented a wavelet tree to reduce the size of genomes in memory.. It was just breathtaking.

fizx• 9 years ago

FWIW, a few years back, I was averaging <5ms for a benchmark of live traffic (many thousands of queries per second) on Pinterest queries against their full dataset on a single EC2 box with a weeks worth of customization on top of Lucene.

The existing stuff isn't slow.

veni08• 9 years ago

Yep, the current index will be anyway in memory, I am writing a branch of the cache disk.

veni08• 9 years ago

The document is improving.

veni08• 9 years ago

The data is a real-time storage disk, the index size is limited by the available hard memory space.

conmarap• 9 years ago

Ah, this is awesome! So far I had to rely on Lucene++, and it could get complicated at times. Using go is something I had been wishing for.

veni08• 9 years ago

Yep, go can be a simple deployment.

wiremine• 9 years ago

Sidebar: I wish open source authors would think a bit harder about naming their projects. Here's some other projects already named riot:

- http://riotjs.com/

- https://riot-os.org/

- https://github.com/vector-im

KitDuncan• 9 years ago

And riot games. Basically the worst name possible for SEO.

KGIII4• 9 years ago

Then again, the language is 'Go.' I am not sure that was the best naming choice and I've heard complaints that it was initially difficult to search for it. My understanding is that 'golang' has helped as a query. So, I guess the situation has improved but I understand it was problematic at first.

To be clear, I've never used the language. I actually dislike programming, though I've decided to get back into it because I have a couple of projects I want to poke at. I'm now deciding between Java and Python.

Maybe I should do an 'ask HN' submission.

allenz• 9 years ago

Also, I use luci.criosweb.ro/riot

veni08• 9 years ago

Good names are some well-known things, no way.

ct520• 9 years ago

Does anyone know anything like this that will search pdf text, or tiff?

mikey_p• 9 years ago

http://tika.apache.org

Which I'm pretty sure can be embedded in Solr, has plugins for Elasticsearch and others.

maxpert• 9 years ago

I wish I could understand mandarin, such nice projects need some good translation.

veni08• 9 years ago

Thanks for understanding, the document is improving.

sAbakumoff2• 9 years ago

I mean, we have bleve[0]. What else do you need, really?

[0] https://github.com/blevesearch/bleve

veni08• 9 years ago

Multi-language and distributed support, simple.

sAbakumoff2• 9 years ago

bleve search does support multiple languages.

billconan• 9 years ago

can this replace elastic search?

I'm looking for a light weight elastic search alternative.

hardwaresofton2• 9 years ago

A little late but if you didn't know you could do reasonably fast Full Text Search with SQLite... Now you know:

https://sqlite.org/fts3.html

https://sqlite.org/fts5.html

burntsushi• 9 years ago

Does it suffer the same limitation as PostgreSQL's fulltext search? i.e., It doesn't use corpus frequencies in its ranking function. (I skimmed the docs but couldn't immediately find my answer.)

hardwaresofton2• 9 years ago

I'm not sure -- but I'm going to guess yes? I tried to look around and couldn't find an answer either...

Riot – Full-text search engine in Go

Related Stories