marcuslager

I bought a license simply beacuse Sublime Text was the only editor that could open a wikipedia json data dump. I installed many editors before I more than willingly paid for Sublime.

I mean, great idea, I'm stealing it.

Edit:

A distributed serch engine: https://github.com/kreeben/dire

Open for feedback.

Stolen!

Now there are too many rules. I'll just go back to writing crap code.

Summarize how it changed my belief system (or gave it a dent)? Or summarize the video? Ok, here's a summary: the video is just one of the most watchable things I've come across.

What you said made me want to write surpisingly terse and concise code. It's what I shall try to do from hereon.

I'm scrolling up to see where you hit a nerve with me (or is it the other way around?). Anyway, sorry about that.

I'm glad I posted here because one point is maybe finally clear to me about speed. It's not a good word for performance. It's the wrong word to use.

The great folks at Elasticsearch would _love_ for Lucene to be more performant. It would make life so much easier for them.

The Lucene team spend a good buck on their nightly performance tests. It's astonishing how well-tested Lucene is.

I wonder why I'm faster at writing and reading. Maybe it's because I have been benchmarking against an older version (4.8). But still. I wonder if my tests are all wrong or if I just got lucky in my design. ResinDB has flaws. It puts massive pressure on GC at writing time, if your batches are huge. I'm working hard at optimizing that achilees heel away. Before I have completely done so, writing speed is achieved through lots of memory allocations. It's surprisingly easy though to move away from using GC as a service.

But performance is not a feature anymore?

Edit:typos and tried to clarify things

You download it and then after some time you unstall it, got it. Well, I don't think I need to worry about MongoDB then ;)

But I know what you mean. ResinDB is a library that let's you embedd a database inside of your application. It's not a service such as MongoDB. MongoDB load and keep indices in-memory. That's the fastest type of architecture you can have if you want to answer quickly to queries. To have it all in-memory.

Well, there _is_ one faster way. It's to construct a smart index file, bitmapped, stored on a SSD, where the data is laid out in such a way that reading from it is just as fast or faster than reading from an in-memory data structure. This is what Resin achieves.

I agree that a library such as this project is not at all as consumer-friendly as an application is. Some might even call it completely unsexy. It's a component of something bigger though, something you can indeed call into from nodejs. But that's another project.

or sell directly to those who know exacly what you are talking about

Yeah I've been thinking I should try to get a few gigs as a speaker at tech meetups or conferences to talk about this tech but I haven't yet found a good enough story to tell.

Edit: give me one more chance to describe what Resin is.

Have you heard of SQL Server LocalDB? It's proper SQL Server, but it runs inside the process of your application. It's a library that has support for SQL, fast reads and writes. It's a database like any other databasem but it's a library.

Unfortunately (or not) SQL Server LocalDB has no support for full-text search. This is why there is a marketplace for libraries such as Lucene, who make full-text search their priority. That market place has been as fixed on Lucene, an open source free software project (LGPL), as the world has been on Google for about as long a timespan. If I want to make a dent on that market I need to be as open as Lucene and as performant.

To keep up with achademia the code base of a search engine should move fast (my view). Managed code lacks the preciseness of C++ but allow you to work fast. As hell.

So, ResinDB looks very much like a much smaller (in code size) version of Lucene. We will see the coming months or so, who moves the fastest. Me or the Lucene team.

Thanks for that very good advise.

From what I hear, folks (not only here at HN) aren't really looking for speed, instead they look for features. I'm starting to look at performance less and less as something to strive for and more and more just a confirmation that the architecture is healthy.

Say my benchmarks are correct and I do in fact beat my closest competitor by some measurement. I'm thinking I should start utilizing that performance to achieve higher relevance, donate it if you will to that cause instead of the speed cause.

To me though, Lucene is not so much of a competitor as she is a role model. Well maybe she's both. My real competitor, some day, will hopefully be Elasticsearch. I've used versions 2 and 5. I'm underwhelmed.

Thank you so much for this feedback. My eyes have been on Elasticsearch ever since their first funding of 80 million bucks. But I have also noticed how Postgres is the only database engineering team that seem to care about full-text search. Their indexing capabilities are just awsome. They have a library of indexing types you can use. They all seem well constructed and so does Postgres (and the team).

A HTTP wrapper. Sure. It's in the backlog. I can push it up a bit.

What is it that you like about ELK the most? The easy-peacy install where you immediately can start writing data, the HTTP JSON API, or something else?

"and not have to run a separate process"

Not at all an unrelated issue to me, but unresolvable at the moment me thinks. To use Resin within the same process as a Go app Resin would have to be a Go library.

I'm glad you posted this because I don't think there is a embedded search engine library for Go, which is both a little funny but could also constitute a business optortunity for a Go programmer.

Would you care to talk a little more about your requirements?

"Groonga is an open-source fulltext search engine and column store."

We seem to be at least cousins. Thx for that link. I will have to get back to you.

Edit:

Groonga seems to be cloud software. ResinDB is a in-process library, not a service.

Put ResinDB behind a service end-point and you have "ResinDB as a service", much more like the Groonga architecture.

Orchestration of read/write in a distributed service-like environment is something that is not solved within the ResinDB codebase. ResinDB is intended to be a component of a distributed database, not a distributed database in itself.

Groonga has been around since 2011. I started on ResinDB last year, in March of 2016.

Groonga make monthly releases. I take long pauses because of my lifestyle.

Groonga is a team of devs. I'm an independent solo dev.

Groonga is unmanaged code. ResinDB is managed code.

My claims are backed up by the code I've spent blood and sweat to create. Disprove me please because I need to know of scenarios that I need to solve that goes into vNext, scenarios were I'm currently not doing great.

Edit: and also: I'm reaching out to you guys not because I want a pat on the back or free PR. I'm looking for advise as to how to move from having unique tech to having a business. Is this the right forum?

Many people need to watch this. It reformatted much of my belief system in a disruptive way. That guy is something else.

Thx for the feedback. What I think you should and hope you already do realise is Lucene is nowhere near maximum performance for full-text search nor is it's relevance. And implementing new scoring routines is a drag in Lucene.

Google is also nowhere near maximum relevance. I like word2vec. That model fits into my world view. I'm going to implement it and then take it further. Hopefully while being funded. If not then it shall be my contribution to the open source space and nothing more.

What would impress _you_?

Me personally I don't think it is impressing of Google to be able to store every web page in existance and to refresh them every other minute. I just don't think that is a good way of spending electricity while we haven't figured out yet how to properly utilize the sun's energy. Google is all-knowing while burning shit-loads of coal. What's impressive about that?

I agree to what you said whole-heartedly and I have to conlude by now that I am so not even close to being a sales person. I don't care for the psychology of a sale. I've been in many. I've only seen one or two beautiful ones.

You wouldn't mind me using your data?

I've seen Bubblehunt and I've played aound a little bit with it. Reach me at github.com/kreeben/resin.

Yes I should probably try to step out of the dev role and see things commercially or through a business utility perspective. My mind however wants me to dive even deeper into the rabbit hole that is search and relevance and the hard problems of concurrent read/write. I do believe I'm aligned well enough already for the database/search market but how to find time for marketing?

Edit: I read your post again and you are saying I need to be orders of magnitude cheeper than my competitors. Does the technical solution (being orders of magnitudes more performant) count in any way?

Yes (it helped) and speed, i.e. querying and indexing performance, for sure is only an USP if you also have relevance.

I'm confident the relevance is as good as or better than Lucene. I especially like my phrase queries and how they seem more relevant compared to that of a Lucene phrase query. The scoring is a half-way implementation of word2vec (in a lot of ways similar to the scoring mechanics of Lucene's tf-idf scheme). I'm aiming for full word2vec implementation in vNext.

I have only my own benchmark tests to tell me I'm faster than Lucene. Which is why I'm contemplating writing a formal proof both of ResinDB's performance and of it's relevance.

My test data has been the English verison of Wikipedia plus Project Gutenberg. I suppose I could publish those indices to the world, as a demo search engine. I don't think a soul would care about a proper searchable Project Gutenberg though. Looking into common crawl now.

I'm a part-time father of two, employed doing tedious unmotivating work, focusing completely on my spare time project. I need some advise as to what the next step should be, if I wanted to make this into a business that I could spend all of my time with, not only nights and weekends. Formal proof? Demo?

Side note: one of the most approachable people in the database building community is Oren Eini, creator of RavenDB. He's reviewing ResinDB on his blog. I've read a preview of the entire series of posts, implemented solutions for the best parts of the critique and just released v2. Blog is here: http://ayende.com/blog

"teams in large organizations"

Could be simplified to just "teams.

These days from what I can see teams in organizations with no ability to devise a sane architecture usually take the microservice route. Management loves this new buzzword. To them it means problems go away.

There has to be way to much friction in the mechanics for that to happen. No?

Mitch. One of the greats. Uncopyable.

You can remove "mistakes" like that from your Google history pretty easily.

How is Google search not already sentiment as a service? It compares page relevance partly from sentiment analysis.

What did you have in mind? A text box and a submit your url button? Or something more elaborate?

There is no such disadvantage. Scoring usually comes after the index (tree) lookup. Once you have terms you can weight them. The trie is just a lookup mechanism.

I'm a n00b and postgres vacuum was news to me. Can I ask, in a postgres cluster with high traffic, how do you manage the nodes? Do you take them offline before starting a vacuum process?

Can I have your email address? I just need to send you something. Nothing fishy.

If they have js enabled they already did.