HN user
There currently isn't a way to filter docs alongside a KNN query, and the dimension support is limited to 1024 (a Lucene limitation) and OpenAI embeddings are 1536 dimensions - also indexing performance is not comparable. Wishing this changes, as they're a good stack for the reasons you state
Disclaimer: founder here
We offer a hosted/managed app search experience on top of Elastic or OpenSearch with appbase.io.
You can leverage the best of Elasticsearch (state-of-the-art) engine while getting Algolia like features like search relevance control plane, query rules, out-of-the-box analytics / insights, caching, UI components across all FE platforms (React, Vue, React Native, Flutter).
Hey HN'ers
We're really excited about this launch, as it has been some time in the making. Our goal with appbase.io is to offer a supercharged Elasticsearch experience - we do this by offering an advanced search relevance control plane that allows a user to create a relevant search with no code. Since this experience is powered under the hood with Elasticsearch - so you aren't limited in the future if you wish to build upon what appbase.io already offers. You can host with us, or run it alongside your existing Elasticsearch cluster.
What can you can do with appbase.io: - Build Auto suggestions (powered by n-grams, edge n-grams), - Highlighting support, - Support for 39 languages as well as support for multi-language search, - Popular suggestions based on analytics, - Set synonyms, - Use rank feature in addition to text relevance to optimize relevance tuning, - Advanced query rules to extend search relevance.
In addition to our no code control plane, one can use our JS UI components (4,000+ Github stars, 1MM+ downloads, actively used by more than 1,000 projects) to design the search experience for web and mobile. We offer these components for React, Vue, React Native and Vanilla JS. You can read more about it over here: https://appbase.io/product/search-ui/
When a search experience is powered with appbase.io, we additionally support collecting telemetry on search and click data and provide analytics/insights on areas to improve search engagement and conversion metrics. Read more over here: https://www.appbase.io/product/analytics/.
Hey HN'ers :wave:
Searchbox is designed with the goal of being a lightweight (16KB to 35KB when minified + gzipped), performance focused library for building search UIs declaratively. We have variants for React, Vue and a headless library that can be used with any framework.
Direct Github link: https://github.com/appbaseio/searchbox
With this release version, Searchbox assumes the backend to be an appbase.io/Elasticsearch server. While there are other libraries that are backend agnostic, making this assumption allows the library to support tracking search analytics, displaying popular suggestions, and configuring advanced search relevance options. Searchbox comes from 3 years of learnings we've had in maintaining ReactiveSearch - one of the most popular search UI libraries out there.
Key benefits of using it:
- Searchbox lets you create versatile search experiences. Think auto suggestions, highlighting, incorporating fuzzy search, voice search, setting search relevance (fields to search on, weights to apply), showing default suggestions, rendering your own UIs.
- In addition, it lets you measure search/click analytics, set query rules, show popular suggestions. Most other searchbox libraries don't offer these capabilities.
- It's built with the open web in mind. Search is designed to work with any design system (Bootstrap/Material/Antd/your own), and it can integrate with any kind of UI components. See a demo here for an integration with React Table: https://codesandbox.io/embed/github/appbaseio/searchbox/tree....
- It supports tree shaking. If the built-in searchbox component were not used, the size of the library comes down to 10KB when minified + gzipped.
Read more over here: https://medium.appbase.io/searchbox-1-0-a-javascript-library...
This pricing change from Algolia makes it easier for low-volume search users or someone who is just getting started to use their enterprise features, but at the same time as the parent mentions, makes it an order of magnitude more expensive for mid-to-high volume users. Imo, this is a mismatch as there is already a trial in place for someone who's just getting started to try those features out.
For anyone interested in creating a search at scale, I would recommend checking us out at appbase.io (founder here :wave:) - we provide relevant search, analytics, and access control for search and are built transparently on top of Elasticsearch. It also doesn't hurt that we author some of the most popular open-source search UI libraries: https://github.com/appbaseio/reactivesearch
Does Insomnia offer hosted documentation of the API collections? That's the feature which keeps me using Postman.
Hey, addressing some of the concerns raised here directly:
- client side query construction basically puts you in a position ..
We recommend using a proxy server for these cases. It's on our roadmap to add first-class support for search templates which would completely prevent query generation to happen client-side.
- it's extremely aggressive in sending lots of queries to elasticsearch ..
This is configurable by using the debounce prop. And the number of queries depend on what components you would like to see updated based on a change in the search query or a facet value, for example.
- the queries it constructs are a bit convoluted and under-use such features as filters ..
We would be happy to address specific scenarios if you're seeing, please raise an issue. That said, I believe we already do this to the extent it's generalizable. And since as a user, you can change the underlying query - it should be addressable by users.
---
To sum up, the DOS and querying part is completely configurable and just comes down to usage. We would love to address any generalizable issues.
There are merits to your point about client-side query generation. They are already addressed for users who are using: 1. Appbase.io - as you can set ACLs and rules (e.g. only allow search requests, max X requests per IP per hour, set max size of Y), 2. For users outside of appbase.io, we recommend using a proxy server like you did to implement an authorization logic.
Once we have first-class support for search templates in ReactiveSearch, this should be effectively addressed.
This is good feedback, thanks :-)
Not really, just seems like the dataset didn't contain any match for the first word there. Counter example: https://i.imgur.com/6Z1m1g1.png.
The way a search would work is configurable via props and you might want to set it differently than how we have with the demo. See the doc for this: https://docs.appbase.io/docs/reactivesearch/vue/search/DataS....
Thanks for the compliment. :-) The UI components would always remain open-source.
We offer a hosted search service at appbase.io which funds this and have very recently started offering it more broadly as a managed platform for any ElasticSearch upstream. It offers a better search development experience, out-of-the-box search/click analytics and more fine-grained security features: https://medium.appbase.io/appbase-io-clusters-a-supercharged....
We've been asked about this a couple of times. It's not a trivial undertaking but it's definitely something on our mind.
I am not sure about the merits of the case to comment on this directly (blog post is sparse on details), but this is certainly unfortunate for the ElasticSearch community.
I want to share Arc - https://github.com/appbaseio/arc, an API gateway for ElasticSearch with security features that we have been actively developing starting this year. It's Apache 2.0 licensed, built in Go and we use it for providing security features for all of our customers. It's not as feature-rich as X-Pack / SearchGuard today, but we're happy to accept any PRs.
Hi HN,
ReactiveSearch is a UI components library for creating powerful search experiences. While we originally created it with React in mind, we have added components for Vue.JS and React Native.
This release is a major rewrite of the library for more extensible component rendering support and adds support for controlled components, state access and also comes with experimental support for voice search, search operators and GraphQL. Here are the release notes for 3.0 - https://github.com/appbaseio/reactivesearch/releases/tag/v3.....
Sharing more context about why we built it here: https://medium.appbase.io/appbase-droid-a-lightweight-androi...
"The more advanced multi-tenant use case – where RediSearch was able to complete 25 million indices in just 201 seconds or ~125K indices/sec, while Elasticsearch crashed after it indexed 921 documents, showing that it was not designed to cope with this level of load." previously stated that "Elasticsearch crashed after 921 indices and just couldn’t cope with this load."
It's hard to mistake documents for indices. Both original and the currently edited statement sound strongly suspect and make me question the benchmarking methodology used. What caused the ES to crash after indexing 921 documents? Why is comparing indexing speeds on a 1-node setup even a legit benchmarking test?
In a production setting, I wouldn't recommend doing ElasticSearch multi-tenancy in this manner. Indexes aren't free.
That's definitely good to read. And gathering from other comments, they have done something similar for Security too. :-)
As someone who is involved in the ecosystem and maintains some open-source projects based on ElasticSearch, I am similarly curious about the move to not build upon existing projects (there could be a good reason, but haven't heard it yet).
Why wasn't ElastAlert used / built upon for alerting? https://github.com/yelp/elastalert
Or why wasn't ElasticSearch SQL used / built upon for SQL? https://github.com/NLPchina/elasticsearch-sql
Or why wasn't SearchGuard used / built upon for Security? https://github.com/floragunncom/search-guard
All of them have Apache 2.0 license.
This is a good project to happen, and kudos to AWS for doing this! However, like some in the thread, I suspect AWS has been building this for a while for their own product and Elastic's license change may have influenced the decision to open-source.
Getting a good out-of-the-box granular security has been a long overdue pain point with ElasticSearch, there have been good alternatives for the other problems it is replicating.
1.) ElastAlert - https://github.com/yelp/elastalert for alerting, 2.) ElasticSearch SQL - https://github.com/NLPchina/elasticsearch-sql for writing SQL queries.
Would very much like to hear from OP why these weren't contributed to instead of creating another alternative.
We've been working on a similar project in Golang and now well may be as good a day to open-source and put it out there: https://github.com/appbaseio/arc. It's an extremely light-weight API gateway for ElasticSearch that at the core comes with an out-of-the-box security system based on users and permissions (set granular ACLs, Rate Limits, TTL) inspired from a proprietary security system we built over the past year - https://appbase.io/features/security/.
Docker cloud offered CI/CD integrations and management features similar to K8S that were proprietary to them. While Docker images themselves are portable, this is a whole different beast.
Given the time it takes to migrate the production stack to another system (one or more months), this won't be enough time for many users to migrate. There is no drop-in replacement. It is pretty abrupt of them to give a 60 day notice.
On a meta thought, I wonder what potentially caused this move. It is/was a pretty decent service.
Curious - How is Red sustained, who currently supports it? Or what does the roadmap look like for this to be a five-year sustainable project?
Edit: They seem to address this down in the blog post, with the creation of a RCT token. What is the incentive for a potential backer to buy this token?
Congrats Rodrigo and team! Great to see the progress!!
We aren't redistributing transporter's source code. Where did you see that?
Addressing to your point directly, the project doesn't claim to be open-source https://github.com/appbaseio/abc#licensing and the same is also mentioned in the blog post. The part of the project that is open-source is available at https://github.com/appbaseio/abc (but isn't relevant to the post here).
The import functionality is based on the transporter project[1], and to set the record straight - we will be adding the acknowledgement for the same in the next binary release[2]. However, we are not redistributing the source as `abc import` isn't open-source. For anyone interested in why we aren't straight up using the transporter project as is, there are changes we have introduced in the 1.) sink functionality, 2.) added adaptors for SQL variants and CSV, and 3.) made a more simpler interface. Going forward, we are more interested in the easiest way to sync <X> to Elasticsearch, which is different from transporter's goal of a generic ETL.
I do appreciate you bringing this up. We're very much just getting this out there and want to do the right thing.
1. and 2. It syncs deltas if you are either using Postgres (9.4 or above) or MongoDB. For other sources like MySQL or MSSQL, it needs to import the whole table each time. The tailing capability is DB dependent, for instance, in MongoDB's case - we use the oplog to do this.
The way it works is after the initial import if you are using the `--tail` option, it listens to further changes via oplog. More on setting oplog access over here - https://github.com/appbaseio/abc/blob/dev/docs/importer/adap...
For Postgres, similarly a replication role and a replication slot needs to be created for decoding the write ahead log. More on that here - https://github.com/appbaseio/abc/blob/dev/docs/importer/adap....
This seems like a good primer on replication slots - https://postgresqlspace.wordpress.com/2015/06/18/replication....
Thanks! We haven't tested it at scale, but this is where we would ideally want it to be at v1. If you would like to take it for a spin, would appreciate any feedback on scaling limits.
It can sync to any ES server. Linking to the import doc which expands on that - https://github.com/appbaseio/abc/blob/dev/docs/appbase/impor....
Hi HN, we created ABC import as a convenient way to sync data source to an Elasticsearch index. It does three things really well imho:
1. A small footprint process / docker container that can index or sync your data source with ES that is operationally simple vs relying on application layer logic,
2. Supports on-the-fly transformations with Javascript, as well as configuration of mappings (so if you want to set a specific analyzer on your Text fields, or set type mappings),
3. Works with a wide variety of sources - Postgres, MySQL, SQL Server, MongoDB, JSON, CSV, Elasticsearch and more coming soon.
Currently, only read support is provided to indexes created in 5.x. As someone who used one of their pre-GA releases, they previously had a flag to enable multiple mapping types.