mschoch

You have to handwash the dish, then put it through the dishwasher, then handwash it when it comes out.

You have never actually used a dishwasher have you.

As a resident of the city and former subscriber. 100% YES, that is better than the current situation.

seriously dude, it's over, go home.

in an actual emergency, you would divert traffic from DCA – the logic here is absurd

google insiders will leak for considerably less, no exploit needed

that piece of shit is dead

haha, the goodwill of ending up on a different black list, and never working again

we're waiting for you Mark!!!!!!!!!!!!!!!!

the USDS is the biggest joke to tech people in DC – just silicon valley more cosplay

Sounds like a great opportunity to teach them about the shitty real world we all live in.

Does everyone in NYC live as scared as you?

I think you meant not non non non non non nonsensical

stood up like a badass

Fireproof engineer here, I come from the backend engineer perspective, so I thought I'd share some of my personal hacks on Fireproof:

Fireplace - tooling to deploy Fireproof apps and sync data across your Tailscale network. Once all the computers you care about are on your tailnet, of course you want all the browsers on the tailnet to easily sync with one another.

Go Implementation - Fireproof bills itself as a realtime database that runs anywhere, and I want to make sure that includes inside your Go applications. This will allow your Go application to become a full-fledged reader/writer of the Fireproof ledger.

I'm excited to see what other people want to build and answer any questions.

Bleve is just a library that provides this functionality, you first have to build an application that uses Bleve, and deploy that.

Bleve does (optionally) support persistence, so reading/writing files is one place it does directly interact with the environment. The environment must support mmap.

There are several projects which support distributed index/search workloads with Bleve. The exact approaches vary, but they all use Bleve to perform node local operations, and coordinating this is done at a higher level by the application.

I suspect I don't understand the terminology you're using in the last question, as Bleve has no training, models or nodes.

We recently added an ASCII folding filter, which may help: https://github.com/blevesearch/bleve/pull/1070

We were made aware of this in September: https://github.com/blevesearch/bleve/issues/783#issuecomment...

First, here are the top two requests that we do NOT plan to implement.

1. Make Bleve a distributed index, or make Bleve into something that is a more direct ES competitor.

We have no plans to do this because we think that is better built at a different layer. We have hooks we introduce in certain places where we need to plug-in code that would otherwise violate the boundaries. And that is an arrangement that has worked well so far. There are multiple projects built on top of bleve that allow you to index/search across nodes.

2. Make an adapter for the XYZ key/value store.

This request goes back to the original bleve index which is serialized into a key/value abstraction layer. When users run into size/speed issues with bleve, many assume that just plugging in a faster key/value store will help. (Hey we thought that too when we built it this way)

But, we've now replaced that index scheme with a new implementation called scorch. Scorch is considerably smaller and faster, and manages it's own index on disk, without using any key/value store.

As for things that we DO plan to implement:

1. Size of the index still comes up a lot. Couchbase is a very performnace sensitive user of Bleve, so I expect they'll lead the way on this front.

2. Better (pluggable) scoring. Today our search result scoring is broken for several types of queries, and the stuff that does score right is too tightly coupled to the searching logic.

3. Overhaul index mapping. Today bleve uses a mapping object to describe how source objects/documents are indexed. One of the best ways we can simplify the mapping is to make things more explicit. I think we tried to embrace the concept of reasonable defaults, but we ended up with inheritance hierarchies that are difficult to reason about.

There are lots of miscellaneous things like adding a data type that supports IPv6, or more advanced queries (lots of variations on span queries).

Bleve originated as part of a solution to a problem customers faced when using Couchbase. Almost all customers have at least some sort of search use case, but often times that use case isn't particularly complicated. Many of them were running an ES cluster, moving the data from Couchbase to ES with an adapter, and using that to solve their search use case.

However, many of those same users complained about having to operate another cluster, especially ones that weren't already using the JVM (since it was a skill set they didn't have).

So, the appeal was to offer a service that runs as a part of the Couchbase cluster. It wouldn't have to match every feature of ES, just shoot for 80/20 and customers would likely find it beneficial.

It was fortunate that Go was still growing in popularity within Couchbase at that time, and we were able to position Bleve as a true open-source component, on top of which some money-making value add could be layered.

Core contributor here, happy to try and answer any questions.

One of the big things we're working on at the moment is improving the release process. In addition to semantic versioning of the APIs we have to think through how it applies to the binary artifacts created. We want Go modules to be supported and be a part of the solution, but we are also mindful not to break things for existing users.

Concurrent queries are supported (not sure what you mean by store being a single file, it is a directory of many files).

Concurrent indexing is also possible, so long as you can arrange to not put duplicate document ids into batches executing concurrently.

As for usage, it is just a library, so it is typically embedded in a single process (though this can serve multiple clients concurrently).

Distributing the index across multiple nodes is done at the application level. At Couchbase we do this with bleve in a separate project called 'cbft'.

https://github.com/couchbase/cbft

You are correct. I think when I originally designed the homepage I wanted to show how you index and search in just a few lines. I thought omitting the error handling boilerplate was acceptable, but properly handling these errors is also key for a good initial experience. So I'm persuaded to at least revisit the decision.

https://github.com/blevesearch/blevesearch.github.io-hugo/is...

With the current index scheme we would regularly do 10s of gigabytes (which is embarrassingly small in our opinion). We haven't done large scale testing with scorch yet, but in some smaller configurations the new index is 1/10th the size of the previous one, so we hope to have moved the bar considerably on data sizes that work.

Yeah, the old index format had many factors contributing to it taking up considerable space. As a single data point, we have a beer-search sample app, this includes a data directory with 29MB of JSON files. In the old index format, with mapping that did a realistic configuration of indexing many fields, and storing some of them, the bleve index size was over 200MB.

With scorch the index size is 22MB. Query performance is comparable (and we haven't even gotten to really tuning this yet).

Yes, when you create a new index, you can choose the index implementation. The previous one was called 'upsidedown', and this one used key/value stores for the actual storage. So, you would also specify BoltDB/RocksDB/LevelDB/Moss/etc. The new index implementation is called 'scorch' and it writes directly to disk instead of going through a key/value store.

Bleve has support for querying across multiple indexes (shards) but does not prescribe any mechanism to split the data. So, it's up the application to divide the data how it sees fit, but you can use Bleve functionality to execute the same query across multiple indexes and merge the results.

Merging is required and is indeed somewhat resource intensive. Bleve's current indexing approach has no segments, instead all index data is serialized into a key/value store. This approach allowed us to experiment and plug-in a variety of implementations. Unfortunately, the key/value abstraction limits the way you interact with data, so there are a number of drawbacks. One key gain we get from the segmented approach vs the key/value store approach is that we no longer need to maintain a backindex to handle updates/deletes.

I was watching one of those engineering disaster shows and thought it would make a good name for a project. I didn't find any other software projects using the name, and it seemed like it would have decent googleability. The relationship to fire/explosions has given good themes for logos and sub-project names (like scorch).

Well since someone submitted us here with no apparent reason or context, allow me to provide something of interest. (primary contributor of bleve here)

Just recently we merged support a new experimental index scheme called 'scorch'. This new index scheme is designed from the ground up to reduce index size and improve performance. It features:

- a segment based approach, much like Lucene

- vellum FTS for the term dictionary - https://github.com/couchbase/vellum

- roaring bitmaps for the postings lists - https://github.com/RoaringBitmap/roaring

- and compressed chunked integer storage for all the posting details

It's still experimental at this point, but shows considerable indexing speedup, index size reduction, and similar query performance to the old index format used today.

The code for this new index scheme can be found here: https://github.com/blevesearch/bleve/tree/master/index/scorc...

Might be using non-standard notation, I just followed what I saw used here: http://blog.burntsushi.net/transducers/

The values after the / are the output values associated with the transition.

Weighted finite state transducers are not supported.

Thanks,

Author here, yes the observation that it is immutable is correct. One approach you can use is to have some lighter-weight representation for the most recent data. Then have one or more FSTs representing the older data. As time permits keep merging the new data and the older ones down into new larger FSTs. Merging them is straightforward since you can iterate the contents in order, which is the order you need to build new ones. In this way, it is very similar to having a WAL up front, and one or more segments backing an LSM storage.