mfreed

github.com/timescale/tigerfs

Hey, author here. Just noticed this.

Yes, TigerFS works well with local PG - in fact, all of my testing framework uses this setup.

Big upside of this is latency; main downside is that such a deployment is primarily "single user mode."

One very interesting approach of this is ability to transparently share state / collaborate across machines, with these transactionally serialized at the common database.

I primarily work with Postgres, including both at scale and in very ephemeral environments. So just haven't looked at all what doing with this SQLite would mean.

None of this. It's in the blog post in a lot of detail =)

The 5ms write latency is because the backend distributed block storage layer is doing synchronous replication to multiple servers for high availability and durability before ack'ing a write. (And this path has not yet been super-performance-optimized for latency, to be honest.)

Currently support multi-AZ clusters and multi-region disaster recovery (continuous PITR between regions).

We're continuing to evaluate demand for multi-region clusters, love to hear from you.

I'm not aware of any published source for this time limit, nor ways to reduce it.

The docs do say, however, "If the volume has been impaired for more than 20 minutes, you can contact the AWS Support Center." [0] which suggests its some expected cleanup/remount interval.

That is, it is something that we regularly encounter when EC2 instances fail, so we were sharing from personal experience.

[0] https://docs.aws.amazon.com/ebs/latest/userguide/work_volume...

Tiger Cloud certainly continues to run on AWS. We have built it to rely on fairly low-level AWS primitives like EC2, EBS, and S3 (as opposed to some of the higher-level service offerings).

Our existing Postgres fleet, which uses EBS for storage, still serves thousands of customers today; nothing has changed there.

What’s new is Fluid Storage, our disaggregated storage layer that currently powers the new free tier (while in beta). In this architecture, the compute nodes running Postgres still access block storage over the network. But instead of that being AWS EBS, it’s our own distributed storage system.

From a hardware standpoint, the servers that make up the Fluid Storage layer are standard EC2 instances with fast local disks.

A few datapoints that might help frame this:

- EBS typically operates in the millisecond range. AWS' own documentation suggests "several milliseconds"; our own experience with EBS is 1-2 ms. Reads/writes to local disk alone are certainly faster, but it's more meaningful to compare this against other forms of network-attached storage.

- If durability matters, async replication isn't really the right baseline for local disk setups. Most production deployments of Postgres/databases rely on synchronous replication -- or "semi-sync," which still waits for at least one or a subset of acknowledgments before committing -- which in the cloud lands you in the single-digit millisecond range for writes again.

TimescaleDB primarily serves operational use cases: Developers building products on top of live data, where you are regularly streaming in fresh data, and you often know what many queries look like a priori, because those are powering your live APIs, dashboards, and product experience.

That's different from a data warehouse or many traditional "OLAP" use cases, where you might dump a big dataset statically, and then people will occasionally do ad-hoc queries against it. This is the big weather dataset file sitting on your desktop that you occasionally query while on holidays.

So it's less about "can you store weather data", but what does that use case look like? How are the queries shaped? Are you saving a single dataset for ad-hoc queries across the entire dataset, or continuously streaming in new data, and aging out or de-prioritizing old data?

In most of the products we serve, customers are often interested in recent data in a very granular format ("shallow and wide"), or longer historical queries along a well defined axis ("deep and narrow").

For example, this is where the benefits of TimescaleDB's segmented columnar compression emerges. It optimizes for those queries which are very common in your application, e.g., an IoT application that groups by or selected by deviceID, crypto/fintech analysis based on the ticker symbol, product analytics based on tenantID, etc.

If you look at Clickbench, what most of the queries say are: Scan ALL the data in your database, and GROUP BY one of the 100 columns in the web analytics logs.

- https://github.com/ClickHouse/ClickBench/blob/main/clickhous...

There are almost no time-predicates in the benchmark that Clickhouse created, but perhaps that is not surprising given it was designed for ad-hoc weblog analytics at Yandex.

So yes, Timescale serves many products today that use weather data, but has made different choices than Clickhouse (or things like DuckDB, pg_analytics, etc) to serve those more operational use cases.

Our experience is that Clickhouse and Timescale are designed for different workloads, and that Timescale is optimized for many of the time-series workloads people use in production:

- https://www.timescale.com/blog/what-is-clickhouse-how-does-i...

Sidenote: Timescale _does_ provide columnar storage. I don't believe that the blog author focused on this as part of insert benchmarks:

- Timescale columnar storage: https://www.timescale.com/blog/building-columnar-compression...

- Timescale query vectorization: https://www.timescale.com/blog/teaching-postgres-new-tricks-...

All depends on the benchmark :)

https://www.timescale.com/blog/what-is-clickhouse-how-does-i...

That PoC benchmark didn't turn on Timescale's columnar compression, which every real deployment uses. So misleading at best.

(Timescaler)

Check out how TimescaleDB adds columnar compression to PostgreSQL, typically saving 95% of storage overhead:

https://www.timescale.com/blog/building-columnar-compression...

https://www.timescale.com/blog/how-we-scaled-postgresql-to-3...

Staring at a >trillion rows in a TimescaleDB hypertable on PostgreSQL.

Also, Timescale similarly introduced S3 for bottomless data tiering:

https://www.timescale.com/blog/expanding-the-boundaries-of-p...

Can you say more about "dynamic labels"? Do you just mean that as you evolve, you want to add a new type of "key-value" pair?

The most common approach here is just to store the step of "dynamic" labels in JSON, which can be evolved arbitrarily.

And we've found that this type of data actually compresses quite well in practice.

Also regarding compression, Timescale supports transparent mutability on compressed data, so you can directly INSERT/UPDATE/UPSERT/DELETE into compressed data. Under the covers, it's doing smart optimizations to manage how it asynchronously maps individual mutations into segment level operations to decompress/recompress.

(Timescale cofounder)

If you are open, would love to hear more about some of the challenges you had with Timescale, esp. with your workload.

mike (at) timescale or DM on twitter?

Do you use AWS Athena as the "database/serving layer" for customer-facing applications where performance matters?

Our experience is that isn't a common use case for Athena, while it is the primary use case for TimescaleDB.

Very much agree.

Even without this new capability of S3-based storage, Timescale's native columnar compression often gets like 95% storage reduction, even while staying fully in the PostgreSQL ecosystem.

(We often see our users amazed about how much less storage/disk they require after migrating from RDS.)

https://docs.timescale.com/timescaledb/latest/how-to-guides/...

This is a great observation. As you point out, this was designed for the workload patterns we typically see with time-series, events, and analytical data, where the query (& insert) patterns differ across time.

So I agree that it's good for cold storage, but it's a bit nuanced. For example, you rarely see small random queries to old historical data, but you do often see larger scans over historical data. And in those cases, the throughput you get from S3 is actually quite high (especially that we've engineered with with proper columnar compression and row group/columnar exclusion). Which is very different from a latency bounded workload where you have a lot of small random reads, which is much more common in CRUD-like workloads.

Also, with Timescale, you have the ability to build continuous aggregates (incrementally materialized views). So you can have the raw data (or even lower levels of rollups) that get tiered into S3, while the more frequently accessed rollups can remain in hot storage.

(Timescale co-founder)

"Can I use Timescale to provide a SaaS service that collects application traces, and I provide a DSL to query the database that is not exposing the DB directly?"

Yes you can. (Timescale co-founder here)

The vast majority of people using TimescaleDB use our community/open-source software (rather than managed Cloud), and vast vast majority of those use the TSL (Community) edition, including many as part of their SaaS service.

It's a three-part test for asking "is this a value added service?" [0] Given the way your have described your service, sounds like a clear "Yes".

- Is your SaaS service primarily different than a database product/service? Yes.

- Is the main value of your SaaS service different than that of a time-series database, and you aren't primarily offering your SaaS service as a time-series database? Yes.

- Are users prevented from directly defining internal table structures through the database DDL? Presumably yes.

[0] https://www.timescale.com/legal/licenses#section-3-10-value-...

TimescaleDB supports compression for all data types, it just uses type-aware compression algorithms that it applies automatically/transparently based on typing.

- Gorilla compression for floats

- Delta-of-delta + Simple-8b with run-length encoding compression for timestamps and other integer-like types

- Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top)

- LZ-based array compression for all other types

https://www.timescale.com/blog/building-columnar-compression...

So as to your question, just turn on compression; it's very common to see 94-97% reduction in storage.

(Timescale co-founder)

We see those types of queries commonly in TimescaleDB. And, for example, both compression and "horizontal" scale out has ways where you can optimize your code for these types of analytical queries.

More concrete, we see a lot of web3/crypto use cases, and making a wallet ID, NFT name, or ticker as a top-level considerations.

E.g., use your contract address as the segmentby field for compression.

I'm not sure what "recent license changes" you are referring to?

The Timescale License was introduced in late 2018, although we never _relicensed_ any of our Apache 2 code, we only created a space for _new_ capabilities to be developed under these terms.

There were a few changes to the license in Sept 2020, but those only liberalized a few clauses (including from feedback we received from the Hacker News community, including the "right to repair" and "right to improve"). At the time, we also moved any remaining "enterprise" features from an Enterprise/Paid license into the TSL/Community/Free license.

https://www.timescale.com/blog/building-open-source-business...

We haven't changed anything since, nor have we ever moved any Apache-2 code into the TSL.

(Timescale cofounder)

We actually do this within Timescale Cloud, and it's amazing.

It allows us to cohort performance data across data stored in others microservice databases (e.g., by account types, projects, billing data, etc.). JOINs across foreign data wrappers using TimescaleDB + Postgres, all within the database and without ETL or application code needed.

So you could look at Prometheus data for your trialers vs. customers, for customers running more than X services, for customers that pay more than $X per month or have been a customer for more than 6 months, etc.

It's super useful across operations, support, product, customer care, and more...

Actually, try https://slack.timescale.com/ instead if you are signing up for the first time.

The link above is the right slack group, but you need to go in through the URL I provided to get the right invite tokens. Sorry, Slack isn't fully designed as a community platform, even though many use it for such.

Yes, we support the rich set of PostgreSQL's JOIN operations, including against hypertables. It's generally smart enough to only apply these JOINs against the right subset of time-series data if you also have any time predicates (due to the way we perform "constraint exclusion" against our hypertable chunks).

There are other common queries related to what you describe, like a "last point query": Tell me the last record for each distinct object. Here, for example, we've built special query optimizations to significantly accelerate such queries:

https://www.timescale.com/blog/how-we-made-distinct-queries-...

To replicate, please see the Time Series Benchmark Suite, which is open-source and has many vendor-contributed configurations:

- https://github.com/timescale/tsbs

- https://github.com/timescale/tsbs/blob/master/docs/timestrea...

Hi! So the team is over 100 at this point, but engineering effort is spread across multiple products at this point.

The core timescaledb repo [0] currently has 10-15 primary engineers, with a few others working on DB hyperfunctions and our function pipelining [1] in a separate extension [2]. I think generally the set of outside folks who contribute to low-level database internals in C is just smaller than other type of projects.

We also have our promscale product [3], which is our observability backend powered by SQL & TimescaleDB.

And then there is Timescale Cloud [4], which is obviously a large engineering effort, most of which does not happen in public repos.

Interested? We're growing the teams aggressively! Fully remote & global.

https://www.timescale.com/careers

--

[0] https://github.com/timescale/timescaledb

[1] https://www.timescale.com/blog/function-pipelines-building-f...

[2] https://github.com/timescale/timescaledb-toolkit

[3] https://github.com/timescale/promscale ; https://github.com/timescale/tobs

[4] https://www.timescale.com/blog/announcing-the-new-timescale-...

Yep!

And in fact, that's exactly what TimescaleDB supports - things like hyperloglog to support approximate count distinct, including as part of continuous aggregates. [0]

This blog post - "How PostgreSQL aggregation works and how it inspired our hyperfunctions’ design" - provides a really nice description of how our the API design of some of our analytical functions are motivated by the ability to "split" processing into the "pre-aggregation" and "finalization" steps, with the blog post focusing on the example of percentile approximation. (I think it was on HN a while back as well.) [1]

[0] https://blog.timescale.com/blog/introducing-hyperfunctions-n...

[1] https://blog.timescale.com/blog/how-postgresql-aggregation-w...

Timescale implements its own compression algorithms. It includes several ones, and automatically applies the choice of algorithm based on the data types of columns.

- Gorilla compression for floats

- Delta-of-delta + Simple-8b with run-length encoding compression for timestamps and other integer-like types

- Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top)

- LZ-based array compression for all other types

This means within even the same table, different columns will be compressed using different algorithms based on their type (or inferred entropy).

More information for those interests:

- General TimescaleDB compression post: https://blog.timescale.com/blog/building-columnar-compressio...

- Deep dive on compression algorithms it employs: https://blog.timescale.com/blog/time-series-compression-algo...