Fergi

From Craft: https://m.youtube.com/watch?v=vBkzm4a7iY4&feature=youtu.be

In 2015 Razorpay pitched themselves as “Stripe for India.” Look at them now!

This is exactly correct. Private blockchains create efficiency between large institutions like JP Morgan and other banks who are constantly moving large volumes of money around by eliminating the traditional processes in place required to move that money. A private blockchain is essentially a shared database (distributed ledger) used by an arbitrary number of private institutions, each of whom has an interest in participating to reap efficiency gains by using a shared database that they can trust, more than they would trust a traditional database owned by a single party. It's not game changing innovation in the same way that crypto futurists think of Bitcoin as a new type of decentralized money, it's incremental efficiency for existing processes. It's more accurate to think about this as blockchain technology for JP Morgan than as cryptocurrency in the sense that Bitcoin is an application of blockchain technology. JPM coin is not like Bitcoin, or ripple for that matter.

The important thing to understand here is that JP Morgan thinks about blockchain technology completely differently than libertarian futurists. Blockchain is not a movement for them - it's a technology that reduces friction for things they are already doing.

Yes, that is my opinion. And if it's possible to build something minimal that helps demo or describe the product to users, it's totally reasonable to build that thing quickly and take it to customers for feedback.

The pitfall to avoid is investing large amounts of time, energy, and money building products in a vacuum based on assumptions about what people want and will pay for. I've heard this referred to as committing "assume-icide."

(Jeff from PipelineDB & post author here)

Thanks for this comment. I think you articulated the thought process that this post aims to speak to beautifully. Builders do want to build, and finding an audience first and doing the type of tedious customer development work described in this post IS an impediment to building, which is precisely the point I wanted to make.

Assuming that the goal of building a product is to ultimately generate revenue, having a temporary impediment between conceptualizing a product idea and building the product is a good thing. This impediment allows for the builder to pause and objectively scrutinize his own idea, using feedback from potential customers as data about the extent to which the product hypothesis is correct.

You're right that stagnation is the worst outcome. And the inverse of stagnation is momentum, which will exist to the extent that people want what we're making for them, something that can be determined in advance of building simply by talking to potential users and customers.

I'll also add here that the process mentioned in this post in no way inhibits the type of creative and inspired thinking that developers use to envision game-changing products. It's quite the opposite - rigorous and merciless scrutiny of our own ideas is the distillation process that allows us to refine our ideas into their essence, then confidently build things with conviction, and be right.

(Jeff from PipelineDB / author here)

Thanks for the honest feedback. You are definitely correct that there are many successful projects and businesses that did not start with a clear sense that people would use or buy their products. But there are a disproportionately large number of projects and businesses that have failed, precisely because they did not have a clear sense that people wanted what they were making and would pay for it.

Also, this post isn't saying that builders shouldn't begin with a strong, fundamental conviction about how things should be dramatically different. I, as the author, would actually argue the exact opposite. The point I'm making is that once we have our convictions, we should test and measure the extent to which those convictions are correct before investing large amounts of time, money, and energy into productizing them.

Lastly, the point about asking questions and then listening to customer feedback would only result in building minimally better products if the product hypothesis we start with is itself unimaginative. But that hypothesis can be literally anything. Customer feedback simply teaches us if market demand lines up with our assumptions about market demand.

I think this is a smart strategy, and is essentially using marketing in the same way that this post suggests using sales, to gauge customer interest in a potential product. The extent to which manual sales outreach or the kind of marketing technique described here should be used likely depends on the kind of product in question. B2B products with higher contract prices and a smaller number of total potential customers will likely require a sales strategy, where B2C products with lots of users and lower contract values may be better served with this kind of marketing tactic.

Great point, though! This is an important topic to consider with marketing and sales strategy development. Thanks for the comment.

(Jeff from PipelineDB / author here)

Good questions!

I would definitely suggest being straightforward and honest with the potential customers you're talking to. And ideally, it would be better to have a MVP you can demo than nothing, but the main point here is that talking to potential customers before building anything is a better strategy than building a product and then talking to customers.

If you talk to a bunch of potential customers about a product idea and discover that there is demand for the product, then building a MVP that you can demo would be a logical next step. And if the MVP is something you can build quickly, then it probably makes sense to do this sooner than later.

The trap to avoid here is building products in a vacuum, based on assumptions about what people want and will pay for. Doing the customer development (pre-sales) work to prove, or disprove, customer demand for a product before building the product is wise.

No, not currently; however, we offer hosted deployments of PipelineDB as well as a SaaS product called Stride (stride.io), which is based on PipelineDB.

Just out of curiosity, do you have a specific use case that necessitates stream-stream JOINs, or were you just exploring the docs and wondering about this?

Fundamentally different approaches. Citus focuses on horizontal scaling for PostgreSQL and PipelineDB focuses on continuous aggregation for large volumes of streaming time-series data.

This only applies to continuous views, not all PG tables. Think of continuous views in PipelineDB as very high throughput, incrementally updated materialized views. Raw data hits continuous queries in PipelineDB (continuous views) and only the output of the continuous queries is stored. So 1 billion events ingested could be distilled down into a single row that incrementally counts up from 1 => 1 billion as each data point arrives, instead of storing all of the 1 billion raw data points and counting them up later.

PostgreSQL 11 support is imminent =)

(Jeff, PipelineDB Co-Founder here) - thanks, Chucky! We appreciate your support!

We are about to release an open-source extension for PostgreSQL that will help with the bloat problem you mentioned here, at least for continuous updates like high volume aggregation (see: pipelinedb.com). PipelineDB will be refactored as a standard PostgreSQL extension this month. Continuous aggregation can help with the bloat problem for use cases where you only need aggregated / summary data, like realtime reporting & monitoring.

Perhaps the most ironic thing I've read in a while:

"Suicides would also decrease, as the most of them are result of depression, which is mostly caused by anxiety about survival."

The storage engine for PipelineDB is PostgreSQL and the output of continuous SQL queries (continuous views in PipelineDB) is stored in what are essentially incrementally updated, realtime tables. You can think of PipelineDB as very high throughput, incrementally updated materialized views, also.

see: http://docs.pipelinedb.com/continuous-views.html

Powering real-time reporting dashboards is definitely the #1 use case we see for PipelineDB from open source users and customers of our new SaaS product powered by PipelineDB, called Stride (stride.io).

Not for streaming analytic workloads, because PipelineDB fundamentally adds continuous queries to PostgreSQL, so data is continuously distilled and aggregated as it arrives, before it is stored, which drastically reduces the amount of data stored in PipelineDB (or soon, in PostgreSQL via the extension refactor).

PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/docs/)

But in terms of ad hoc, exploratory analytics workloads, yes - the scaling limitations would be the same, since for ad hoc, exploratory analytics PipelineDB and PostgreSQL are the same. But with that said, the processed, aggregated data that gets stored is generally much smaller than large volumes of granular data, so there is much less data to comb through with PipelineDB.

PipelineDB will become a standard, open-source PostgreSQL extension this year, enabling anybody using PostgreSQL to leverage continuous SQL queries in combination with all other PostgreSQL functionality.

https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-...

Jeff from PipelineDB here. We have been rolling Stride out in developer preview in waves, prioritizing the strongest use cases and customers with the biggest need and immediate budget for their projects. We've had a pretty massive backlog of demand for Stride so apologize for the delay in getting everybody who wants access involved in the developer preview. We will keep you updated via email and hope you understand that we are also working hard on the open-source development of PipelineDB and supporting PipelineDB Cluster customers while we continue to improve Stride.

We apologize for not communicating better about the Stride timeline and will do a better job of that moving forward.

Feel free to email me at jeff (at) pipelinedb (dot) com with specific questions or to discuss this further.

Thanks again for your patience and understanding here.

Jeff from PipelineDB here. Yes, PipelineDB is an analytics product designed to continuously query high volumes of streaming data. The top use cases we see are building realtime reporting dashboards and realtime monitoring and alerting systems.

PipelineDB excels at workloads where you know the queries you want to run ahead of time, where your workload fits within the confines of SQL, and where there is a high degree of distillation (aggregations, sliding windows, etc.). We have large customers including Charter Cable / Time Warner Cable, MediaMath, SmartNews, MOAT Analytics, Cradlepoint Networks, Paddy Power Betfair, and others using the system at scale and we offer a clustered edition of PipelineDB under a commercial license, but PipelineDB's single server edition is open-source.

We have a live chat room here for technical questions:

https://gitter.im/pipelinedb/pipelinedb

Jeff from PipelineDB here. We are also rolling out a SaaS product based on PipelineDB called Stride (stride.io) that enables this type of computation via an API with little to no overhead.

Jeff from PipelineDB here. Our products are similar in that they execute SQL queries continuously on streaming data, but the main differences are that PipelineDB is 1) open-source, and 2) includes a fully functional relational database, which enables "state" and doesn't presume that data is intended to be streamed out as well as in. The latter is a seemingly subtle, but important difference because "stream out only" drastically limits use cases.

I'd also add that PipelineDB excels at SQL-based workloads where you know the queries you want to run in advance and want to stream large volumes of data into PipelineDB's continuous query engine and store ONLY the results of the continuous queries in PipelineDB's underlying relational database. The main value is in continuous computation and distillation for realtime reporting and realtime monitoring and alerting use cases. PipelineDB isn't designed to do ad hoc, exploratory queries, although it can to the same extent that PostgreSQL 9.5 can. It's designed for scenarios where you know the analytic queries you want to run in advance and where a SQL-based approach to streaming analytics with integrated storage provides value by being simpler than building a custom system in Java / Scala using frameworks like Storm, Spark Streaming, Druid, Cassandra / HBase, etc.

The main difference between PipelineDB and old CEP frameworks is that PipelineDB is based on the PostgreSQL 9.4 core, so is a fully functional relational database in addition to a streaming processing engine, which gives us integrated storage in addition to a SQL-based approach to stream processing and realtime/streaming analytics. We have also added functionality like probabilistic data structures, stream-table joins for adding historical context to streams, and now offer a distributed architecture, realtime alerting, and HA. So there is definitely some overlap with the types of things CEP frameworks did, but having an open-source core based on PostgreSQL that operates with 100% SQL enables a wider range of users to deal with an increasing need to tackle realtime data challenges.

PipelineDB (YC W14) | Jr. Systems Engineer and Sr. Frontend Engineer | San Francisco - ONSITE only | Full time | jobs@pipelinedb.com

We're on a mission to build a new type of database for a modern world in which information is constantly moving, and moving fast. PipelineDB runs SQL queries continuously on large volumes of streaming data, giving companies the capability to easily develop scalable, realtime applications and services using only a familiar SQL interface. No application code is required. This inherently involves solving a lot of big problems, many of which are novel. We’re looking for creative engineers who appreciate the value and freedom of choosing their own projects, approaches, and working with other top talent in a low distraction, streamlined work environment. Our small team has backgrounds from Berkeley, MIT, Facebook, Locu and AdRoll, and we're all doing exactly what we want to be doing: building a groundbreaking new product out of thin air. As an early stage engineer you'll ultimately own a very large part of the product. Which part of the product you take charge of depends on where your interests are, but there are several different potential areas of focus. You'll be entrusted to make sound architectural decisions as well as implement your vision effectively. We are well funded by top investors including SV Angel, Susa Ventures, Data Collective, Paul Buchheit, and more. If you’ve been waiting for an opportunity like this, please send your resume and a quick blurb about yourself to jobs@pipelinedb.com.

Benefits: * Full medical/dental/vision insurance * No set work hours--work when you feel smart * Choose your own setup * No vacation policy other than that it is strongly encouraged * Large equity ownership

Jeff (PipelineDB Co-Founder, here) - Yes, PipelineDB is great for this use case. One powerful aspect of PipelineDB is that it is a fully functional relational database (a superset of PostgreSQL 9.4) in addition to a streaming-SQL engine we have integrated the notion of 'state' into stream processing, for use cases exactly like this.

You can do anything with PipelineDB that you can do with PostgreSQL 9.4, but with the addition of continuous SQL queries, sliding windows, probabilistic data structures, uniques counting, and stream-table JOINs (what you're looking for here, I believe.)

PipelineDB is actually licensed under the GPLv3. We accidentally updated our files with the AGPL earlier today by mistake but have since rectified that. Apologies for the mixup!

I'm not familiar with riemann.io but the main differences seem to be that Riemann is clojure-based and would require external storage of some sort. PipelineDB is SQL-based and has integrated storage. It really depends on what kind of event processing you're looking to do.

The main tradeoff with PipelineDB and other stream processing frameworks like Riemann, Storm, Spark Streaming, Samza, and others is mainly flexibility for simplicity. Not all streaming computation lends itself to SQL, but in scenarios where it does continuous SQL queries and a relational database can be simpler. But as with all data processing endeavors, you have to find the right tool for the job.