tango12

TL;DR: one Claude per slack channel.

How is this multiplayer? Claude-slackbot needs to inherit my permissions for accessing scoped context and data / tools. Sounds like I need to setup specific credentials for each instance of the agent.

More importantly, claude-slackbot automatically remembering sounds like it'll be company wide AI slop? 75% of the stuff on slack should not be remembered for the future. And 90% of the stuff worth having in context is not even on the slack thread.

The AI being wrong problem is probably not insurmountable.

Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed.

Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not.

If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.

Author here.

You’re right in that it’s obviously not the only problem.

But without solving this seems like no matter how good the models get it’ll never be enough.

Or, yes, the biggest research breakthrough we need is reliable calibrated confidence. And that’ll allow existing models as they are to become spectacularly more useful.

What’s the eventual goal of text to sql?

Is it to build a copilot for a data analyst or to get business insight without going through an analyst?

If it’s the latter - then imho no amount of text to sql sophistication will solve the problem because it’s impossible for a non analyst to understand if the sql is correct or sufficient.

These don’t seem like text2sql problems:

Why did we hit only 80% of our daily ecommmerce transaction yesterday?

Why is customer acquisition cost trending up?

Why was the campaign in NYC worse than the same in SF?

Counter-opinion:

I totally agree with the sentiment here, but disagree with the conclusion.

1. The rest of the world doesn't have facebooks data layer. Ergo, GraphQL is very hard for everyone who's not FB.

2. Turns out GraphQL is actually a really good data API. Whether it's a postgres/mongo/graph/REST data source. Because it's a great mid-point between something as flexible as SQL but as controlled as a REST API.

Here are 3 things that GraphQL is absolutely great at:

  - Select a 2 fields out of a 100 attributes in a data model? GraphQL is great.
  - Gradually deprecate an old field in a database and replace with a new one? GraphQL is great.
  - Want a flexible way to filter/join/paginate regardless of the underlying storage layer? GraphQL is great.

Working with GraphQL ought to feel like you're working entirely with your database and sprinking the right bits of transform/validation/authz business logic and then the API that other teams can use is "free".

GraphQL is dying as a replacement to a REST API layer. I think GraphQL will see its second wind as a data API. Microsoft and Google announced their GraphQL data APIs recently.

GraphQL will probably only make sense when it's as close as a 1:1 map to a DB beneath it. Whether relational or NoSQL or graph.

In all other cases, the GraphQL juice is not worth the squeeze.

I’m the founder of Hasura - sharing some notes from how I’ve seen GraphQL usage evolve over the last few years.

1. GraphQL was and remains insanely hard to build without an underlying data layer that does the heavy lifting. Without projection push-down, predicate push-down, a data layer that can support heavy parallelism it’s untenable. Exactly the problems the OP highlights - needing to “hoist”…

2. GraphQL on REST is an anti-pattern. The hype takes you there, but the juice is not worth the squeeze. The problem was lack of types? Add openapi. The problem was custom aggregation endpoints? Make it super easy / cheap to build aggregate endpoints in REST. Use an AI copilot to write an aggregate REST endpoint and a openapi schema for it even. But maybe the last thing to do is to build annd mainatian another API layer with a completely different execution model.

It’s not what it was meant for perhaps, but GraphQL’s killer use-case is an API to access / operate on data. Especially when there are multiple consumers of data (services, apps) and they don’t own the underlying data sources or speak the underlying language of the database. This turns out to be the underlying problem behind a lot of api, data modernization / migration / decomposition efforts.

The cost of this approach is implementing Graphql by building a compiler/planner with a robust authz system baked into it. A resolver based approach can never cut it, unless you resolve a query plan - aka build a compiler.

If you want to use graphql to just batch rest endpoints, it’s very likely going to become legacy tech as soon as the team that built it starts to move on.

Love this!

Curious, wouldn’t this be great for folks with internal APIs as well?

Team 1 can consume team 2’s API with an SDK instead of a raw REST client.

I think there are 2 subtleties here on where you optimize perf:

1. Frontend makes n+1 requests to the api. Either use GraphQL or build a custom endpoint that shapes a perfect response.

2. Build an api server that fetches data from the db efficiently without n+1 hitting the DB.

The whole tension introduced with resolver/data-loader ways of building GraphQL, is that while it makes 1 easy, it makes 2 harder. Compared to bespoke REST-ish endpoints.

@alex_lav: Your SQL query here is exactly what I meant! In SQL, this is easy, both to write and to optimize perf.

Implementing this with GraphQL resolvers (how folks typically write GraphQL servers) is hard - there's a users function and an articles function that are both called, so it's hard to implement those 2 functions contributing together towards a query plan. This was my point.

In fact a REST endpoint that takes a list of users and runs this query with the `IN` part parameterized is easier to build here. But the GraphQL version is a pain.

Like you said, it's not the join cost that's significant :)

But for the DB to be sending less data over the network means that the DB is doing less CPU work in helping process that data.

Benchmarking should be straightforward for a particular use-case.

The problem is that it’s hard to get the articles resolver make one query to fetch all the recent articles. So you’ll end up with n+1 fetches from the db.

In the articles resolver, you have to:

Select from articles order by created where article.user_id = X limit 5.

And you’ll have to run this statement n times - once for each user.

Not sure how a data-fetching cache layer at the app server layer will help here.

it's often much easier to scale the application server layer that the DB layer on most cases.

Hm…ignoring the specific design decision here on using json agg - I think the open question is - is that specific sentiment practically still the truth in 2023? Is it worth scaling app servers or is it better to use Postgres well and scale a managed PG offering?

Do we need to spend time writing boring code when we could instead just use DBs better? And then instead we can spend time where it matters in the app server.

And in most situations you want to "expand" the data as close as possible to the final location.

What do you mean by “expand”?

From a perf pov: Imagine you’re fetching 100 authors and each had 1000 articles, would you rather a) fetch mxn Cartesian product or b) ask the DB to send you a json aggregation where each author has a 1000 articles? There’s a significant db perf, intermediate layer ser/de, and network latency difference right?

query {

  Users (first: x) {

    …,

    articles (first: y, sortBy: CREATED) {

    ..

    }

  }

How would one implement the users and articles resolvers - that would be as efficient as a most_recent_views in sql?

True - but the important point worth discussing here is that most common GraphQL implementations make it really hard to think in joins.

The resolver pattern and data loader pattern make joins really painful.

eg: implement a simple top-N query (fetch 10 users and the most recent 5 articles for each) in GraphQL with a resolver tree, or even data loader.

Totally feel the pain. Ironically, most DBs have started supported really good json aggregation functions.

Combined with lateral joins, it’s possible to even push this ser/de to the DB directly and have minimal unpacking and packing in sending that data back to the client!

IMHO: The way we often approach GraphQL execution is a little dated. It’s perhaps legacy thinking from times when we had data layers that couldn’t be expected to scale even for the majority of use cases. But that has completely changed over the last 5-10 years - DBs can scale pretty massively, for specific workloads with very little work required at the app server layer. So if we used DBs to actually model as close to our domain entities as possible (and yes, we’ll need multiple DBs sooner or later), then a huge part of our work is already technically done.

So - here’s another way to approach building GraphQL:

1. Build a “read graph” so that you get a highly composable API that can almost entirely be pushed down into a data layer (eg: Postgres). This would involve techniques similar to building a compiler. Which is hard, but the great news is that it’s easy to make reusable across people who share the same data layer (Eg: Postgres).

There are only 2 business logic concerns at layer, both of which can be pushed down, which helps simplify the problem. 1) modelling/transformation: which can be done in the language of the data layer and pushed down (like views) 2) authz: which can also be done in the language of the data layer and pushed down (like rls)

2. The “command graph”: a set of controllers written in whatever language you please that makes changes to the data layer (or delegates to an orchestration system) and returns a reference to the read graph after having made changes.

This is kind of how we approach the problem at Hasura - a connector SDK that makes it easy to build reusable query compilers for different DBs or API services.

And it’s been great to see similar ideas in the ecosystem - let’s make it easy to execute a tree of functions of create a query plan, instead of executing a tree of data fetch functions (like the OP). Grafast by the postgraphile folks comes to mind.

(From hasura)

Came across this blogpost that happens to describe where something like hasura sits and how it decouples product dev (app + bff) from data api (microservice + db) dev.

Think of Hasura as a GraphQL BFD (backend for data) instead of a GraphQL BFF.

Given the complexity vs DX trade off with GraphQL presented in the post: perhaps GraphQL goes the way of SQL.

SQL is a phenomenal API but nobody builds a “sql api server” at work.

People do build products that present a sql api, because it’s a great API.

(from Hasura)

Scaling subscriptions is hard, but we work with our users/customers at scale to make sure settings are tweaked correctly.

We have users running 100k - 1M concurrent users in production for live-event type platforms. It's not completely trivial to benchmark and setup because query patterns, streaming vs live queries etc have an impact, but it works very reliably. No missing events, no problems disconnecting/reconnecting, no need for sticky sessions and so on.

An initial POC benchmark [1] should be a quick affair so if you're trying it out and run into any problems, please hit me up! Email on my bio.

[1] https://github.com/hasura/graphql-bench

Hasura CEO here. Totally hear you.

We're working on a new serverless infrastructure layer that'll make the pricing better for users compared to a DIY API server or to a self-hosted Hasura.

It's a significant engineering lift on our side - at its core we're engineering Hasura to achieve 90%+ infrastructure utlization (no cold-start, sub-millisecond auto-scaling), and that's what will allow us to do this.

Not there yet, but we'll be demo-ing and talking about the engineering at HasuraCon in June!

Hm…ideally your remote schema can return one or more product ids and then you can create a relationship back to the model from Postgres.

Link to docs: https://hasura.io/docs/latest/remote-schemas/remote-relation...

Hasura CEO here. If you're trying or evaluating using hasura (or nhost) along with Yugabyte / Cockroach feel free to reach out (email in bio). Happy to have our team help with any architecture or benchmarking things you might run into!

Haven’t looked at the internals or the API, but I think for starters, Hasura is designed for working on any Postgres including ones that have existing data, especially modern Postgres offerings (crunchy, alloy db, neon, Aurora, cosmos Postgres, timescale, yugabyte)

Hasura supports multiple data-sources and creating relationships across multiple data sources.

Sources can be databases (postgres family, sql-server, big-query currently, and as we add support for new databases) or REST APIs or GraphQL APIs.

This composition allows in a final GraphQL API that federates across different sources.

From our docs: https://hasura.io/docs/latest/data-federation/data-federatio...

Yep. Exactly this.

Our focus (Hasura) is on the last mile so that innovations on the data side (eg: materialize, ksql, timescale continuous aggregates) are “just obvious” to start building applications and services against.

Has anyone come across neat tools for load-testing streaming APIs?

We used https://github.com/hasura/graphql-bench and a set of scripts to monitor runtime characteristics of Hasura and Postgres, and reconciliation to make sure data was received as expected and in-order.

But would love to see if there's other tools that folks have come across!

Streaming data to clients at the edge (apps or services) is a hard problem. We built an approach that keeps Postgres at the center and allows clients to consume a stream of data that is already in Postgres without any additional moving parts.

This works with read replica style scaling or with Postgres flavours that support scaling out easily (eg: Cosmos Postgres, Yugabyte & cockroach coming soon).

We recently added a streaming GraphQL API to Hasura to allow clients to easily consume data from Postgres as a stream.

There are solutions to ingest and store a large amount of data or a stream of data. However, once this data has been captured, securely exposing this data as a continuous stream to a large number of HTTP clients concurrently is a challenge. And that’s where streaming GraphQL API fits in.

Would love to see benchmarks! :)

But just for clarity in the discussion thread here, Hasura also compiles to a single query when only Postgres is being hit and I'd expect performance to be quite similar...

Ofcourse, if the GraphQL query requires federation across multiple Postgres databases or multiple databases or databases + other GraphQL / REST APIs, then Hasura breaks them up into multiple queries with a worst-case performance of a data-loader type set up.