philippemnoel

There is significant work happening in the PostgreSQL ecosystem to make "use a single system to run everything" possible. ParadeDB is one such system pushing on full-text/vector search, with some light aggregations, at the index level. There is also work being done by DuckDB folks with pg_duckdb and others like Xata. (disclaimer: I work at ParadeDB)

Let's go. Very bullish on PgDog. Lev understands this space better than anyone else. If you are sharding Postgres, you should talk to him.

I've been wanting this for so long. This is so clever.

Slightly concerned Apple would patch this by preventing others from using the same bundle identifier as their official apps, though?

Writing code was never the job. Writing code was always a means to the job. This is why people hire "software engineers", not merely "programmers".

ParadeDB maintainer here :). We would happily make it available on Azure (and all other cloud providers!) if there were a way for us to earn a living in doing so.

Fyi, we are in discussion with some hyperscalers on making this possible.

ParadeDB (YC S23, https://paradedb.com) | Database Internals Engineers, Distributed Systems Engineers | Remote or hybrid in San Francisco | Full-time

ParadeDB is a Postgres extension that delivers state-of-the-art search in Postgres. Our customers, including Alibaba Cloud, BILT Rewards, Modern Treasury (YC S18), and many others, rely on us to deliver the performance and feature set of dedicated vector databases/Elasticsearch natively in Postgres, simplifying their stack and day-to-day operations.

If you like systems, this might be for you. We're a team of 10, a Series A company based in SF but distributed across North America. We're hiring for database internals work (e.g., building the database in Rust) and distributed systems work (e.g., building the database-as-a-service platform in Go).

We're open-source at https://github.com/paradedb/paradedb.

Our technical blog: https://www.paradedb.com/blog

More info on roles: https://paradedb.notion.site/

This a 100 times^. The Postgres ecosystem is remarkable and has managed to strike the balance between OSS and commercial successes in a way that most infra verticals have not.

We (ParadeDB) use Barman via CloudNativePG for almost all our deployments. It's been solid, although I've had a few complaints about 1) inability to set S3 storage classes, 2) slow upload for very large databases.

Nonetheless, very happy to see this project on the front page of HN!

ParadeDB (YC S23, https://paradedb.com) | Database Internals Engineers | Remote or hybrid in San Francisco | Full-time

ParadeDB is a Postgres extension that delivers state-of-the-art search in Postgres. Our customers, including Alibaba Cloud, BILT Rewards, Modern Treasury (YC S18) and many others, rely on us to deliver them the performance and feature set of dedicated vector databases/Elasticsearch natively in Postgres, to simplify their stack and day-to-day operations.

If you like systems, this might be for you. We're a team of 11, Series A company based in SF but distributed across North America.

We're open source at: https://github.com/paradedb/paradedb. Our technical blog: https://www.paradedb.com/blog More info on roles: https://paradedb.notion.site/

https://github.com/paradedb/paradedb -- Full-text & vector search natively in Postgres

Good catch, thank you both -- fixed!

Under the hood, ParadeDB is built by integrating Tantivy (a Lucene-inspired Rust search library) inside Postgres. This is to say: I agree with you -- We're not trying to claim that Postgres itself is an alternative to Lucene, but rather that something like Lucene can be integrated inside Postgres so that you can get the power of both in a single system (or in a cluster of Postgres instances)

(ParadeDB maintainer here) Yes! We've already built some faster analytics in Postgres, and have a lot more coming. Here's some relevant context in case you're curious: https://www.paradedb.com/blog/faceting

Anytime! We have some vector search work coming in the next few weeks/months that I expect you'll find interesting. Stay tuned :)

We just made a blueprint for it! https://github.com/paradedb/render-blueprint

One-click deploy with Render, and we're directly in contact with the core team to get it added to their official docs. I hear the PR is up internally :)

(ParadeDB maintainer here). This is super cool. Congrats on the project, and I'm excited to see ParadeDB be used to power this kind of use case. If there's anything else you need to ship Omni, don't hesitate to reach out to me!

That's true. For this reason, most modern search engines support language-aware stemming and tokenization. Popular tokenizers for CJK languages include Lindera and Jieba.

We (ParadeDB) use a search library called Tantivy under the hood, which supports stemming in Finnish, Danish and many other languages: https://docs.paradedb.com/documentation/token-filters/stemmi...

ParadeDB | https://paradedb.com | SF Onsite + Remote | Full-Time | Rust Database Engineers

ParadeDB is an alternative to Elasticsearch built on Postgres. We're building a Postgres extension in Rust that offers a new index type optimized for full-text search and aggregate/analytics workloads. We solve three problems with Elasticsearch today:

- Lack of read-after-writes guarantees

- Lack of JOINs

- Infrastructure complexity & cost due to syncing Postgres and Elastic.

We're open-source, and our repository is available at https://github.com/paradedb/paradedb. We're a Series A team of 8 distributed across the US and Canada. Most folks on our team have 10+ years of experience in database internals at companies like Twitter, MongoDB, Oracle, Instacart, etc.

You can find our roles and the profiles of our team members here: https://paradedb.notion.site

If you know Rust and/or have experience working on DB internals and want to work on cool systems problems, shoot us a note. We hire with conviction, have lots of room to grow, and exciting technical problems to solve. My email is phil@paradedb.com.

How so? Many popular projects are AGPL. MinIO, Grafana, etc.

We wrote about this here: https://www.paradedb.com/blog/agpl

The value prop for customers vs Elasticsearch are:

- ACID w/ JOINs

- Real-time indexing under UPDATE-heavy workloads. Instacart wrote about this, they had to move away from Elasticsearch during COVID because of this problem: https://tech.instacart.com/how-instacart-built-a-modern-sear...

Beyond these two benefits, then the added benefits are:

- Infrastructure simplification (no need for ETL)

- Lower costs

Speaking the wire protocol is nice, but it's not worth much.

Bear with me, this will be a bit of a longer answer. Today, there are two topologies under which people deploy ParadeDB.

- <some managed Postgres service> + ParadeDB. Frequently, customers already use a managed Postgres (e.g. AWS RDS) and want ParadeDB. In that world, they maintain their managed Postgres service and deploy a Kubernetes cluster running ParadeDB on the side, with one primary instance and some number of replicas. The AWS RDS primary sends data to the ParadeDB primary via logical replication. You can see a diagram here: https://docs.paradedb.com/deploy/byoc

In this topology, the OLTP and search/OLAP workloads are fully isolated from each other. You have two clusters, but you don't need a third-party ETL service since they're both "just Postgres".

- <self-hosted Postgres> + ParadeDB. Some customers, typically larger ones, prefer to self-host Postgres and want to install our Postgres extension directly. The extension is installed in their primary Postgres, and the CREATE INDEX commands must be issued on the primary; however, they may route reads only to a subset of the read replicas in their cluster.

In this topology, all writes could be directed to the primary, all OLTP read queries could be routed to a pool of read replicas, and all search/OLAP queries could be directed to another subset of replicas.

Both are completely reasonable approaches and depend on the workload. Hope this helps :)

You don't need to. Customers usually deploy us on a standalone replica(s) on their Postgres cluster. If a query were to take it down, it would only take down the replica(s) dedicated to ParadeDB, leaving the primary and all other read replicas dedicated to OLTP safe.

Our customers typically deploy ParadeDB in a primary-replicas topology, with one primary Postgres node and 2 or more read replicas, depending on read volume. Queries are executed on a single node today, yes.

We have plans to eventually support distributed queries.

Yes, Figma!

One of the ParadeDB maintainers here -- Being PostgreSQL wire protocol compatible is very different from being built inside Postgres on top of the Postgres pages, which is what ParadeDB does. You still need the "T" in ETL, e.g. transforming data from your source into the format of the sink (in your example CrateDB). This is where ETL costs and brittleness come into play.

You can read more about it here: https://www.paradedb.com/blog/block_storage_part_one

Postgres is not typically considered to "scale well," but oftentimes this is a statement about its tablespaces more than anything; it has foreign data[4] API, which is how you extend Postgres as single point-of-consumption, foregoing some transactional guarantees in the process. This is how pg_analytics[5] brings DuckDB to Postgres, or how Steampipe[6] similarly exposes many Cloud and SaaS applications. Depending on where you stand on this, the so-called alternative SQL engines may seem like moving in the wrong direction. Shrug.

Maintainer of pg_analytics (now part of pg_search) here. I 100% agree that the statements against Postgres are often exaggerated. In practice, we see both the smallest and the largest companies "just use Postgres" while mid-scale companies often overthink their solution.

That said, there are indeed phenomenal "alternate" SQL engines. I've seen many users see great success on tools like ClickHouse, which ParadeDB is not yet competitive with, and sometimes (dare I say) even Elasticsearch. As for whether this one is one of them... That I couldn't say

Hi folks, ParadeDB author here. We had benchmarks, but they were super outdated. We just made new ones, and will soon make a biiiig announcement with big new benchmarks. You can see some existing benchmarks vs Lucene here: https://www.paradedb.com/blog/case_study_alibaba

This comparison isn't super fair -- ParadeDB does not have compatibility issues with Postgres and rather is directly integrated into Postgres block storage, query planner, and query executor

That's true. We have some more ideas for DataFusion in the works, though... Stay tuned!

Acquisitions don't necessarily mean the end of innovation. Sometimes, it allows them to take innovations they've worked hard on for years and expand the reach to a significantly larger audience :)

I have met the founders of all 3 of these companies and can assure you they all care tremendously about bringing their work to the world.

ParadeDB is independent and without plans to sell anytime soon, though :)

pg_search dev here -- Thanks for mentioning us.

Re: Tantivy. I'm hopeful the community Paul and the Quickwit team have built on top of Tantivy will continue to flourish. I'm sure Datadog will build product(s) with Quickwit, which is built on Tantivy and will contribute to it. Many other companies like ours (ParadeDB) and other databases also integrate it. I can't speak for others, but we'll contribute whenever possible. We're currently working on supporting nested documents in Tantivy, for example, and hoping to upstream this work.

While it's reasonable to be concerned, I'd say this is a win for Quickwit, Tantivy and, of course, the well-deserving team behind them.