chasers

Not a free car because I paid them.

Funny enough their dashboard showed a payment so I just screenshot that for them to look up in their own db!

Rivian lost my full payment for an R1S.

btw recently cleaned up my wal cache busting code quite a bit if you're interested.

https://github.com/Logflare/logflare/blob/main/lib/logflare/...

Need to make a lib out of this!!

I've helped with this one. Happy to answer any questions also.

We use :telemetry to collect usage data per tenant for Supabase Realtime.

We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.

We just had some orphaned database connections (our bug) to a tenant and were able to find and kill them because of Erlang's observability:

https://gist.github.com/chasers/e4aa33625e6ea39ecb89f03383dd...

edit: link to gist

Yo :D This is what Supabase Realtime does!

https://github.com/supabase/realtime

Spin up a Supabase database and then subscribe to changes with WebSockets.

You can play with it here once you have a db: https://realtime.supabase.com/inspector/new

supabase dev here...

this was a very fun load test to run. happy to answer any questions.

shoutout to José Valim and the Dashbit team for the help on Supavisor. elixir and erlang are amazing tools.

connection limits, even with pgbouncer, have been a big support burden for us. supavisor should let us be more liberal with connections we allot to folks.

supavisor is rolled out and available for all projects to use today. contact support[0] and we can expose the supavisor connection string in your dashboard. we'll be proactively exposing it to everyone gradually over the next ~1 month. we'll run pgbouncer and supavisor both for you for probably ~5 months before you need to switch to supavisor.

also very excited at the potential features a custom postgres proxy can unlock.

[0] https://supabase.com/dashboard/support/new

yes otel across all of Supabase in on our radar for sure. we just added ingest support for otel payloads to Logflare (docs coming soon) so when we have that you'll get them on the platform and locally.

if you haven't seen the metrics endpoint we do have an endpoint you can scrape for all your Supabase metrics, and we just improved the example repo quite a bit on how to ship those somewhere: https://github.com/supabase/grafana-agent-fly-example/

although ensuring that only a single client is writing to a database at any time is (mostly) left to the application

Sounds like trouble.

Yeah as in zero ops needed. Maybe NoOps.

I hadn't seen WalEx. Looks rad!!

Phoenix.PubSub is basically a noop service. It really just works.

If discovering nodes is difficult in your env, try using a listen/notify libcluster strategy:

https://github.com/supabase/supavisor/blob/main/lib/cluster/...

Triggers can affect throughput quite a bit on busy tables.

And we didn't want people schemas polluted with triggers.

But also we use Erlang distribution and Phoenix.PubSub because with a global network clients connected to the same node in the same region will get normal Broadcast messages to each other faster. If we ran them through Postgres or Redis the added latency wouldn't work for a global thing.

Accepting pull requests if anyone is interested :D

Happy Livebook user here!

Congrats on all the new stuff this week. Really great work.

Also fwiw, if this is self-hosted I'm sure you can export them from BigQuery in an acceptable format.

And if this is Supabase hosted, you can reach out to support and we can do that for you.

Log drains is high on the list for the hosted platform too.

I work on logs at Supabase.

What format(s) would you need?

For sure > https://github.com/supabase/supavisor/issues/69

Appreciate the detailed feedback here.

Frankly these are all things I've wanted to address. We've just had to prioritize other tasks.

We're definitely going to keep iterating.

Thanks for taking the time to try it!

Wow man Quickwit looks amazing. Probably would have reached for it when I started Logflare if it was around.

I have not seen anything open source that really separates compute and storage like that in one package. You can setup Clickhouse to use an object store but there's a bunch of nuance there. People are starting to hack together something similar using DuckDB on Lambda, basically as a query engine on top of S3. I wonder if you can use DuckDb to get a SQL interface going?

Going to have to take a close look for sure. SQL is definitely a blocker. With Supabase we give people the ability to query their logs with full SQL. And with Logflare Endpoints you create an API endpoint with a SQL query.

Depends on your use case. If the volume of data you're looking at is not much, Sentry is more feature rich and refined.

If you want to expose a logging interface to your customers and/or easily integrate large volumes of structured event type data into the rest of your infra, then maybe look at Logflare.

Yes! You can sign up for Logflare separately actually.

Can you point out ways in which Logflare uses it that makes it so (for ex, is it tiered-storage with a BQ front-end)?

After 3 months BigQuery storage ends up being about half the cost of object storage if you use partitioned tables and don't edit the data.

How does Logflare's approach contrast with other entrants like axiom.co/99 who are leveraging blob stores (Cloudflare R2) for storage and serverless for querying for lower costs?

Haven't really looked at their arch but BigQuery kind of does that for us.

Multiple pluggable storage/query backends (like Clickhouse) is all good, but is there a default that Logflare is going to recommend / settle on?

tbd

Are there plans to go beyond just APM with Logflare (like metrics and traces, for instance)?

Yes. You can send any JSON payload to Logflare and it will simply handle it. Official open telemetry support is coming, but it should just work if your library can send it over as JSON. And you can send it metrics.

I guess, at some level, this product signals a move away from Postgres-for-everything stance?

Postgres will last you a very long time usually but at some point with lots of this kind of data you'll really want to use an OLAP store.

With Supabase Wrappers you'll be able to easily access your analytics store from Postgres.

https://supabase.com/blog/postgres-foreign-data-wrappers-rus...

Curious how was it awful for you? Really have not regretted any part of using BigQuery.

I’m thinking can just use :erlang.phash2 to pin a client connection process to a db connection process. The only thing is a slow query could block a db connection for others pinned to it.

Elixir's Ecto can (and that Go example) so if you use Elixir (or Go) you can use prepared statements with PgBouncer (or Supavisor!) in transaction mode.

Ah `binary_parameters` is just this Go lib thing. Ecto just sends unnamed prepared statements where (TiL) query plans are not cached.

That doesn't say anything about named vs unnamed. This does though:

"If successfully created, a named prepared-statement object lasts till the end of the current session, unless explicitly destroyed. An unnamed prepared statement lasts only until the next Parse statement specifying the unnamed statement as destination is issued."

https://www.postgresql.org/docs/current/protocol-flow.html#P...

Not yet but we definitely are going to do that.

I will say that we are trying to keep the connection to the database open as long as possible and only disconnect when necessary. PgBouncer's db connections are more dynamic.

So if Postgres is doing query plan caching by session then caches would be built up as that query hits other db connections. In theory, this should be better.