Pulsar: Real-Time Analytics at Scale

http://www.ebaytechblog.com/2015/02/23/announcing-pulsar-real-time-analytics-at-scale/#.VOw1CFOsXi8

by mreddy • 11 years ago

54 14 11 years ago

ebaytechblog.com

endymi0n2• 11 years ago

Any insight on how this one compares to Netflix's Druid? Really curious.

pirsquare• 11 years ago

Anyway Druid is by MetaMarkets, Netflix is one of the users

endymi0n2• 11 years ago

Thanks, I stand corrected.

salex89• 11 years ago

They complement each other. Druid is "only" a real-time data storage for events and fast aggregation. This is a complex event processing engine, which would store its stuff in druid.

bmh1004• 11 years ago

Could anyone comment on whether it would be possible to use a system like this for producing on-demand aggregations with filters over 100M rows with sub-second response time?

ddorian43• 11 years ago

From a glance, you have to define the queries before. Are you using anything currently?

What is your insert/update/delete rate? If low, maybe look at postgresql in-memory column-store https://github.com/knizhnik/imcs/

Another way may be voltdb. Since it's partitioned+single-thread/partition, and cross-partition queries are slow, you'll probably have to make a query for each partition and group results on the app.

Also curious.

bmh1004• 11 years ago

I specialize in an in-memory analytics (business intelligence) platform called "QlikView" or "Qlik". It operates on top of RDBMSs/operational databases, so it batch processes data on a daily or hourly basis usually.

syncerr• 11 years ago

Could just use a search db (e.g., elasticsearch) depending on the structure of your data and the aggregations you need.

bmh1004• 11 years ago

For the sake of my curiosity, I am assuming structured data with aggregations like: min, max, average, sorting by another field, count, count unique, etc. I do a lot of work with structured data and predefined aggregation functions, but dynamic filtering.

gianm• 11 years ago

The post says Pulsar can use Druid as a metrics store, so that workload should be doable. Druid is meant for exactly that sort of thing (fast aggregates with ad-hoc filters).

djupblue• 11 years ago

Looks interesting, a bit similar to Apache Storm https://storm.apache.org/

Pulsar: Real-Time Analytics at Scale

Related Stories