Time Series Density Plot

https://observablehq.com/@twitter/time-series-density-plot

by luu • 5 years ago

40 16 5 years ago

loehnsberg• 5 years ago

Aren’t these spaghetti just plots with transparency. Or am I missing something? Another nice way to illustrate density are fan charts. While spaghetti plots show individual time series, fan charts just show quantiles. Spaghetti plots therefore also reveal information of the conditional distribution of a time series which fan charts do not. The latter are just easier to read.

yurivish6• 5 years ago

Hi, author here. You can think of density plots as a way to visualize the data traditionally visualized with line charts or spaghetti plots, but in a way that scales more effectively to large datasets.

Traditional graphics pipelines aren't built to handle more than 256 incremental levels of opacity since it's an 8-bit channel, and when you have a thousand time series you can no longer distinguish between levels of density. Conditional information (following a single line) becomes difficult as well since there are so many lines. Density plots solve the overdraw problem by accumulating density into an offscreen buffer before rendering, which allows the color scale to be tuned to the amount of overlap in the specific dataset. Interactive selection/hover techniques can be used to recover conditional information.

This example is part of a density plot library published earlier this week (https://observablehq.com/@twitter/density-plot-introduction?...) and one of the references there is to an opacity-based approach I've seen used for 2D point clouds, though I haven't seen an analogous demo for curves: https://observablehq.com/@rreusser/selecting-the-right-opaci...

cozzyd3• 5 years ago

so... a finely-binned colormap?

I'm not sure it's not quite right either, because the weight of each series in a bin should depend on the derivative but it doesn't look like it does.

This sort of plot always has a conflict between showing the distribution of each time domain sample and showing the time behavior of each curve. For the former, showing quantile contours may be clearer (though not as pretty!). For the latter different colors with transparency might be better, though it becomes unintelligible at some point.

yurivish6• 5 years ago

The weight does depend on the derivative: https://arxiv.org/abs/1808.06019

(This is mentioned in the intro post for this library: https://observablehq.com/@twitter/density-plot-introduction?...)

cozzyd3• 5 years ago

hmm, it's not obvious (to me!) in the single-line case, but that might be because of the high-frequency noise that makes the derivative sort of random.

floatrock2• 5 years ago

Wasn't immediately clear to me why the derivative comes into play, but let me see if I understand the gist of it. I think this is something interesting I haven't thought about in dataviz before...

So in a traditional 2D heatmap, we discretize the space into squares or bins, then color the bin "proportional" to the number of points that fall inside the bin.

If we're making a 2D heatmap of a timeseries, though, we don't really want to count up the discrete points that make up the timeseries because our binning may not line up with our timeseries sampling frequency. Especially not if we want a very high-resolution heatmap, where each bin is sized to be 1 pixel.

So the solution isn't to count the number of points (from the timeseries data series) in each bin, but rather to count the number of lines that get projected/drawn into each bin (pixel).

But we need to go a step further... if we think of each bin as a hypothetical square, the shortest line through a square is a vertical or horizontal line. And the longest line through a square is a diagonal line. So if we want to represent "how much" of the line goes through a square, we need to measure its slope. Hence, derivative.

So by 2D heatmapping lines instead of points, we're not just counting how many lines fall into each bin, but we also need to weigh each bin-line occurrence by it's arc length, which we use the derivative as a proxy for.

Neat! (I think...)

saltypal• 5 years ago

We use a homegrown gnuplot version of this at $work, and I can speak from experience that you can immediately spot problems (timeouts, as a set of ghostly horizontal lines, or outlier requests) that would be absolutely hidden in a traditional p50/p95/p99 chart.

I wish more (hosted) monitoring systems would shift to this type of visualization, since computers are fast now (I'm told!)

tantalor• 5 years ago

"Here's how to..." call a function in a library that does all the work for you.

Expected a bit more.

yurivish6• 5 years ago

This page was specifically created as a library example. Here's the full collection: https://observablehq.com/collection/@twitter/density

If you're interested in the technique behind the plot you can read more about how that works in this paper: https://arxiv.org/abs/1808.06019

atemerev• 5 years ago

One of my favorite plot types. Very simple and immediately conveying the necessary information. My colleagues in the lab dislike it, though, they prefer something more traditional with error bars, as they are more quantitatively inclined.

mighdoll• 5 years ago

Nice work! Density plots aren't just pretty. I think they show a more accurate story for any time series plot with more than a few thousand points.

The reduceOp caught my eye in your API. What uses have you imagined/tried for changing the density accumulation function from x + y?

yurivish6• 5 years ago

Thanks! I've used it when the data is itself binned frequency data, which is one way to reduce data volume (e.g. binning 2 weeks of minutely time series data into (hour, percentile) buckets and plotting them as weighted points).

When the number of the bins in the data is not an exact integer multiple of the number of bins in the histogram, adjacent data bins can get mapped to the same histogram bin, resulting in e.g. 2x the data volume in some rows/columns of the histogram.

When a bit of loss of fidelity is acceptable the two solutions I've used are to render the histogram at an exact factor of the number of data bins then set the canvas dimensions to the desired size (relying on the browser to downsample the resulting image), or to use `max` as the reduceOp and render the histogram directly at the intended size.

Time Series Density Plot

Related Stories