wuputah

I think it will be straightforward to expose time_bucket in pg_duckdb. Feel free to open an issue for the feature.

That is an incorrect and baseless accusation, we had nothing to do with "Postgres (tuned)". My commits are only in the `hydra` folder. There are no restrictions on how you set up the benchmark in Clickbench and the settings we use there are analogous with what we use on our cloud service for a similar sized instance.

As the linked post points out, the main 'advantage' of the "tuned" benchmark is the indexes, which are tuned specifically to the queries in the benchmark. We do not use indexes in our version of the benchmark, aside from the primary key (which actually provides no performance advantage).

does it work with the existing postgres apt/yum repos?

We only support apt for now but plan to support other package managers in the future. It works with existing Postgres apt packages, we recommend using PGDG but the default system packages on Debian/Ubuntu work as well.

Does it work with the postgres Docker image?

yes, in fact this how our `container` feature works. https://docs.pgxman.com/container

In my opinion, we plan on accomplishing this by using a container; it's not quite something we have today, but this is good feedback. :)

On Ubuntu/Debian, Postgres doesn't typically work this way, so it's not the way that pgxman works. pgxman works on top of the existing `postgresql` packages and with the existing package manager (apt) in order to install extensions -- which is also how it handles runtime dependencies, whether libraries or even other extensions.

So, that said, we have a container feature I could see using to effectively isolate for a single project. Right now there is only one single "global" container (per Postgres version) that pgxman will manage for you, but this is just a MVP of this feature. I could definitely see something like `pgxman c dev` or similar which will read a local pgxman pack file (pgxman.yaml) in your project and boot a "local" Postgres for you just for that project.

The pgxman pack is already a thing and is how the local container config is maintained, but we haven't tied it together in the way described above... yet. For more on both pgxman pack and the container feature, check out our docs.

Thanks for calling these out, as these are just misunderstandings. We will certainly tweak the language around these.

- Installing the extension itself does not change the default table type, this is only the case on Hydra Cloud and our Docker image.

- "Hydra is not a fork" refers to the fact that Hydra did not fork Postgres; it is an extension. We have put in a lot of effort since forking Citus, but it's not our intent to hide that fact.

- Yes, "Hydra External Tables" is a productization around FDWs, there's more we want to do with it but it hasn't been our focus lately.

First we added a bitmask to mark rows as deleted - these rows are filtered out on read. Then updates are implemented as deletions + inserts. We have also added vacuum functions to remove/rewrite stripes that have >20% of deleted rows in order to reclaim space and optimize those stripes.

of course :) drop by our Discord if there's something you'd like to contribute and want to chat about it beforehand, need help/have questions getting started, etc. https://hydra.so/discord

Yes, Hydra Columnar and PostGIS get along just fine. We've not looked into any PostGIS-specific optimizations yet, but if users run into issues, we'll be happy to investigate.

Of course, that's the power of Postgres. You can join between columnar tables or between columnar and heap (row-based) tables. The performance of joins hasn't been a specific focus of our engineering work yet, but I made a little test of enriching an analytical query with user data here: https://gist.github.com/wuputah/e62b83f86880bd7e6623809afe4c...

Yeah, I agree! However, ClickBench has used 500GB GP2 as the "standard" for some time, so I stuck to it for consistency. We use GP3 for our hosted service, and I did test on GP3 as well with identical settings as GP2 and the results are very similar.

The metadata can act as a basic form of indexing (or sometimes caching, though Hydra doesn't use metadata to calculate results yet), but it's not an index in the traditional sense. It's used to eliminate stripes and blocks from consideration during a scan.

Columnar is not ideal for a `users` table where you want to select and update specific rows, often in very small, quick transactions (OLTP). You would want to continue to use a traditional (heap) table in that case. That's certainly something you can still do with Hydra, and combining both kinds of tables is considered HTAP, and something that is a unique use case of our product.

To contrast, columnar is best for "fact" tables -- data about something that happened (thus it does not change) that will be analyzed in an aggregate way. Those might be logs, events, transactions, etc.

A narrow distinction, but Hydra is Postgres - we only install an extension - while Greenplum and Redshift are forks but remain Postgres-compatible (to varying degrees). I'm not up on when Greenplum last merged updates from Postgres, but I would be concerned that it only runs on Ubuntu 18.04. If you have a look at the Greenplum install in ClickBench[1], you'll see it's not a typical Postgres setup. Hopefully we will be able to beat Greenplum straight-up soon. :)

Redshift is multi-node, which puts it in a different category -- with considerably higher costs.

[1] https://github.com/ClickHouse/ClickBench/blob/main/greenplum...

We hope you find our new product useful and engaging! You can get started with a free 10 GB Hydra at https://hydras.io/ or read our blog post at https://hydras.io/blog/2022-03-21-announcing-hydra-postgres-....

I agree with your lead statement and argued as such, but was overruled. Last I knew and understood, Heroku Status is static pages pushed out to Fastly, with the internal admin site (that does that work) running in a Heroku Private Space. If you look at the DNS, it still appears to be served by Fastly, and Heroku Private Spaces are generally pretty isolated infra, so I would be curious what the failure mode was here. But ultimately this is the fire you play with when you self-host your status site...

Hi! Definitely reach out to us to discuss further.

Certainly, I see us expanding to other cloud providers as we follow customer demand, but it will take some time. I think if you wanted to move faster and have a higher level of control, self-hosted would be the way to go. We are offering that but it's not on our web site.

Definitely stay tuned [0] for Clickhouse! And yes, exactly, you can continue to use your ORM of choice.

0: social links are at the bottom of hydras.io :)

Hi, JD here, CTO at Hydra. In an HTAP scenario, local transactional data would be replicated, but your data warehouse will likely have a great amount of data that your Postgres database does not. You can still connect that data to Postgres with Hydra. Ultimately, it's up to you if/how you choose to replicate your data -- along with guidance from our team along the way.

Hi, JD here, Hydra's CTO. It's still early days and we are considering open source; for now, we wanted to leave our options open, and OSS feels like a one-way door. I think you make a great point here - thanks for sharing your past pain / experience. Definitely food for thought.

Our "no lock-in" claim refers to your data, since Hydra is Postgres, you're not stuck using "HydraDB" forever -- it's relatively easy to migrate in or out since you can use well established Postgres tools. We also are open to licensing the product should you wish to self-host, on-prem, etc.

Hi, JD here, Hydra's CTO. Thanks for the interest and questions!

Today, queries need to be Postgres-compatible to be intelligently routed, but queries with specific query syntax or functions beyond Postgres can be routed with our manual router[1]. This is our first solution to this problem and plan to iterate in response to customer pain.

Sorry for the confusion! Data moves asynchronously -- we're not trying to implement multi-phase commits -- but we can act on data very quickly once committed. Our solution here uses Postgres logical replication. Using the Data Bridge is optional and a customer's existing solutions are welcome as well.

[1] https://hydras-io.notion.site/Router-a91f5282f1354c54a9ba894...

Hydra doesn't ship data to the client in order to then do further work like aggregations -- that's the whole point of Hydra -- but that also means that you won't be able to "workaround" a performance issue with an underlying data store. For that, we'd need to find a way to replicate the data to a data store that can solve the aggregation performance issue.

Calcite is definitely some cool tech. I can see why it would be attractive for bigger teams, but it seems like a big lift for smaller teams. Our goal is to make it easy for devs already familiar with Postgres to be able to use add databases without learning new tools or adding software... besides adding Hydra, of course!

Absolutely! Hydra is 100% Postgres and supports any existing Postgres-compatible tools.

Hi, I'm JD, Hydra's CTO. There's no perceivable overhead to using Hydra on queries being routed to Postgres. I think you would not be able to see Hydra in the noise in a benchmark -- but it's a great idea to demonstrate this! I will do a blog post! :) Of course, if you were to use Hydra Instance (where your Postgres database is remote) then there will be some network latency.

Hi, JD here, Hydra's CTO. Thanks for the great question!

You can use our manual router[1] to route queries that use a specific syntax or functions. The way this works today is you wrap your query in an SQL function. In the future, we could detect use of a specific features and route those queries appropriately. I think there might be other ways to solve this as well e.g. by having a 'stub' aggregate function in Postgres for the function you want to call. We are working with customers to iterate on issues like this as they occur.

[1]: see "Manual Routing" at https://hydras-io.notion.site/Router-a91f5282f1354c54a9ba894...

Hi! JD here, Hydra's CTO. NoSQL is certainly a challenge but we have a few ideas/angles on how to solve it. Certainly we plan to start with simple queries and then iterate from there based on what our customers need.

We are really excited about the prospect of bringing SQL and NoSQL together!

Hi, I wrote a response [https://news.ycombinator.com/item?id=30443033] about FDWs, I hope it answers your questions.

We will definitely think about S3 support! Would love to understand those use cases more and how Hydra could help.

Hi! JD here, Hydra's CTO.

Hydra does not use FDWs except for Postgres-to-Postgres communication (for now). What we found was that FDWs do not do pushdown very well, even when Postgres has full information. You can get FDWs to push down aggregations, but complex queries with subqueries etc quickly get slow again. In short, our goal is to have your queries take full advantage of the power of each datastores, and we found that FDWs do not accomplish that goal.

We want to support GraphQL at some point, so same goes for us!

We are thinking about an OSS version, I think how we do "limited" is a big part of what that means. What would you like to see in an OSS version? What would you use it for?

I am--was--an EY customer as well for some time, about a year, and had been following it since its inception, particularly through a friend who was a customer while it was still in beta.

1. I haven't seen any significant changes to their product, except for the CLI tool as a replacement for capistrano. The CLI tool is a great improvement, but it's still basically a glorified capistrano tool, and doesn't even get close to comparing to the Heroku command-line tool. Predominately, I feel like the product has been stagnant. Support for Ruby 1.9 and Rubinius (currently in beta) is the most exciting thing I've seen lately. It's surprising rbx has taken so long when it's an EY sponsored project!

Part of the challenge with a product like AppCloud is there is no obvious way to update the stack without disturbing applications, but I'd still have liked a way to potentially update our base software (libraries and such) to something more current.

2. I'm so-so on this. Heroku support is free, but is not 24/7 unless you are a very large customer with a support contract. We paid for support, and EY support was sometimes helpful, sometimes not.

3. I'm not talking about objective comparisons here, these are "shots across the bow" so to speak. I know the pros and cons of both services very well, but what I am talking about are "subtle jabs" about how Heroku is not production-ready/capable, etc.

Yes, all platforms that are static and curated in this way are aging every day, and Heroku has some flaws in this area as well. But as I understand it, the base platform of AppCloud is predominately a Gentoo-based custom portage tree that Ezra built years ago, and it has seen very few updates. This causes problems building gems or bugs in gems that are quite rare since most people do not use software/libraries that are so old.

While there may be amazing things in the works, it also seems to take a long time for features to be rolled out, and often these are in betas initially which do not fall under the support contracts. So if we wanted to move to Ruby 1.9, it's at our own risk (or was, last time I looked).

As for the jabs at Heroku, I have seen quite a few, honestly, but I don't have a pile of links for you. Many of them are subtle or implied, and not necessarily untrue. For instance, sure, Heroku was acquired by Salesforce, but I haven't seen any indication as to why that should influence my hosting decision. Companies are bought and sold all the time. The "ey-migrate" tool was also a great time to get a few subtle jabs in.

Agreed; after using their product for many months, I learned how it is little better than running your own instances. There are some cool things it does, but you end up doing plenty of sysadmin work yourself, writing Chef recipes for anything not supported by their (aging) platform, and then you lose the benefit of support as well. Did I mention that support costs start at $275 (12/5) or $475 (24/7) a month?[1] Without it, you can't open a ticket.

It's also increasingly annoying to see shots taken at Heroku in their blog posts and on Twitter. It's unprofessional. Compete on features and services, please.

[1] http://www.engineyard.com/products/appcloud/support