Update on InfluxDB Clustering, High Availability and Monetization

https://influxdata.com/blog/update-on-influxdb-clustering-high-availability-and-monetization/

by KyleBrandt • 10 years ago

124 70 10 years ago

jrv6• 10 years ago

This makes me think: an open-source project can be better off if it's not controlled by any one company. While in Prometheus (http://prometheus.io), we might still take a while before we have a clustered remote long-term storage, we'd never prevent it because the project is independent of any company and we'd want the open-source project to be as good as it can be.

Also, there were some tentative thoughts about using InfluxDB as the main long-term storage backend for Prometheus, but that has become pretty much uninteresting now that clustering support (needed for LTS and durability) is basically cancelled for the open-source version.

Still, I guess I can understand that when you're a company, you need to focus on making money.

mindcrime• 10 years ago

This is concerning to me, not because I use InfluxDB, but because my company is also a "pure play" OSS vendor. So it's always concerning to see a company fail to achieve success with the pure-play model, and wind up having to go "open core".

True, but consider this: Given that InfluxDB (core) is Open Source, there's nothing to stop somebody else from coming along and building a version with clustering support. Whether that will happen or not is, of course, an open question.

jrv6• 10 years ago

True, but a fork is a fork is a fork. It would have to be done very competently (clustering is hard) and maintained long-term, by people who are not even part of InfluxData and thus have a big starting disadvantage in experience and user base. That makes it much less viable, unless there's a huge critical momentum around it.

bbrazil2• 10 years ago

There's nothing to stop somebody else from coming along and building a version with clustering support. Whether that will happen or not is, of course, an open question.

Clustering is really really hard. My personal experience with that class of problem is that it took 3 years to get it production ready, and that was for a simpler problem than what needs to be solved here.

The chances of a 3rd party implementing this given that InfluxData plan to have a commercial version seem very low. I'd expect that most companies who want to stay open source will treat this as a standard database that needs client-side sharding and HA (which is the recommend approach - see https://influxdata.com/high-availability/).

bogomipz4• 10 years ago

+1 to Prometheus. I'm a new user but have found the project to be well thought out and have found the community on IRC channel to incredibly helpful and responsive to the community. I'm glad you didn't hitch your wagon to Influx. Is there a reason to not continue with your LevelDB backend long term?

jrv6• 10 years ago

Thanks!

We only use LevelDB for Prometheus's time series indexes, while using a custom-built storage engine for the actual series data. But Prometheus's local storage has always been built as single-node storage without clustering - very much on purpose actually, because we want monitoring and alerting to be decoupled and autonomous enough to not have any hard distributed storage dependencies (as that has a high risk of breaking down when shit hits the fan).

We definitely want to support full remote, distributed, and durable storage for Prometheus, but we still want local nodes to have enough autonomous storage for alerting and short-term inspection of data. Whether some of the technology of the local storage engine will be used for building a clustered storage remains to be seen, but it's likely.

troyk2• 10 years ago

I understand the need for businesses to make money, but I do not understand how a business can understand OSS and implement the OSS core-only model. If your product has demand, at some point, an OSS project will arise to displace you.

Are there many winners with this model to name. Maybe NGINX, MongoDB?

eknkc• 10 years ago

- Varnish comes to my mind. Has Varnish Plus with paid features. - There's GitLab with CE and EE editions. - Elasticsearch has a similar approach I think. - As you mentioned, there is MongoDB, NGINX etc.

I think this is a reasonable model. It's not easy to build / sustain complex software and while it works for a lot of projects, maybe majority of projects get neglected or depends on a few dedicated contributors. Who often burns out etc.

It also provides some confidence to users (at least to me). I know the project is open source. If the backing company goes evil, a fork can always emerge (iojs comes to mind). And I also know that some people are making money over this software and they will keep it alive / work on it as long as it sustains the business.

bbrazil2• 10 years ago

Sidekiq is a good example, the author talks about it at https://changelog.com/92/

seiji• 10 years ago

I do not understand how a business can understand OSS and implement the OSS core-only model

Whenever you see "open core," you should read it as "VCs said we need to do this to keep them happy."

VCs love the idea of how you can give part of something away for free then lock people in to auto-renewing subscription contracts for the whole banana (with fees increasing 7% to 17% every year for ongoing "maintenance"). Many VCs still actively recoil at the thought of giving away everything for free then just hoping the best will happen.

Nginx is a weird example because I'm not sure how well they are doing or who would buy "nginx pro" instead of haproxy+nginx or haproxy+varnish+nginx some other combination of already existing stuff.

MongoDB is an interesting example because they went with the "provide open software, but with really serious, game-stopping flaws, then force people to pay for support because nothing works" route. That's almost literally bait-and-switch (bait-and-support?). But, that hasn't ended up so well for them. From a recent WSJ article: Fidelity has cut its valuation of MongoDB in eight of the nine quarters since Fidelity made its investment in December 2013, valuing the shares 58% below what it paid.

troyk2• 10 years ago

Right, so the business model is not new, but winning with it would be.

bogomipz4• 10 years ago

Isn't this what Elasticsearch is doing now? Security is deliberately left out of the offering unless you purchase a very expensive support contract that give you access to Shield? I'm not sure I would call them a winner, its unclear how well they are doing. I think they are mostly running on VC money at the moment. I agree "Open Core" should be regarded with some suspicion.

Do you have a link to the MongoDB WSJ article? They are pretty terrible, even with a support contract bugs have a way of not being resolved or addressed.

KyleBrandt3• 10 years ago

I'm interested in what the pricing will be like at various scales, on Twitter the CEO (Paul Dix) indicated $400 month will be the basic offering for limited cores.

I guess time will tell, but a bigger deal than the money to me is will the clustering actually work well? Clustered databases is an extremely difficult problem. Seen it get better in the years in things like MsSQL, but even with that sort of resources it took a long time (years) for the newer availability model to become stable.

At Stack Overflow we are still using OpenTSBD behind Bosun. But HBase sucks to manage if you don't have any other reason to be using the technology. So I see a lot of users interested in using InfluxDB as the backend (and some do, Bosun can talk to it) because it is easier to get started. But if you know you will have to scale up eventually, all the options right now are not appealing in TSDB land :-/ So if some $$ really does get a good TSDB that scales and reasonable to manage then great, but I'm skeptical.

pauldix12• 10 years ago

We're committed to making the clustering a world class product. You're right, it'll take time. However, we'll get there. We'll put the same focused effort and testing into it over the coming months and years that we put into the TSM storage engine, which is working remarkably well at significant loads.

In the meantime, the stuff we have going into the open source is going to make the standalone server (and thus anyone using open source options for HA or clustering) even more performant, stable, and scalable.

KyleBrandt3• 10 years ago

While skeptical, I am also very hopeful! I think at Stack we will want to set up a cluster and see how it works for us once it becomes available.

We use a relay (bosun.org/cmd/tsdbrelay) to send data to OpenTSDB and it also "tees" it to Bosun for indexing and another OpenTSDB cluster. So sending all of our data to an Influx cluster will be as simple as adding that as another endpoint to our relay since you have an OpenTSDB endpoint.

jrv6• 10 years ago

...anyone using open source options for HA or clustering...

What open-source option for clustering would this be if the main InfluxDB open-source version won't have clustering anymore? A fork or something built on top of multiple standalone servers somehow?

kev0096• 10 years ago

The suck of running HBase is that it is several large components, but it actually works. A product that doesn't work and over promises like InfluxDB sucks infinitely more. Use GCE's BigTable if you want to shed the support of HBase.

meirelles2• 10 years ago

Check out KairosDB. Run on top of Cassandra. It's pretty stable, easy to maintain and update. Comes with some limitations, but still worthy to check out if fits your usage. The server itself is stateless which is great for HA and scaling.

bogomipz4• 10 years ago

I don't believe Kairos has the ability to create any kind of a retention policy - example "after 4 weeks do rollups of the data to hourly resolution, and keep for 6 months total" Can you explain how you handle that with KairosDB? Also it looks like the KairosDB docs haven't been updated in any significant way in a couple of years(other than moving it to github.io) For a project thats been around for a few years this is kind of surprising.

damm• 10 years ago

I really don't see a point to worry about. So as of 0.12 the OSS product will change; however a new product will come out to bring in replication. Somehow they will produce a private binary that will do the same thing

Single node usage is great for most people; sure it can be useful to cluster and you have that option.

If there's limitations it's open source; those can be removed. Then people would likely use the fork that doesn't have that limitation.

I don't see evil here. Just trying to work on their product

jrv6• 10 years ago

I don't think anyone is saying anything about it being evil, and it may even make sense for InfluxData the company. But for users, not having clustering (for scalability, long-term storage, and durability) will limit the value of the open-source version a lot in the long-term. As for the viability of a fork, see the other comments around that.

23david• 10 years ago

I've advocated for and implemented several InfluxDB installations in production over the last year+, and one of the considerations was always that non-alpha (prod-ready) clustering was always promised in the 'next version' that was just around the corner.

Several months ago it seemed clear that the team was overly optimistic, and it's just disappointing to see that now the clustering will be available only in a paid (minimum $400!) option or on their hosted service.

I understand the business considerations here, but it feels like a bait n' switch for all the people who evaluated/used InfluxDB in single-node operation as a temporary measure while giving the team ample time to work out the clustering kinks.

Lesson learned I guess... but dang what an expensive lesson.

pauldix12• 10 years ago

I'm sorry for the disappointment and understand your frustration. Our hope is that for users for whom a commercial solution isn't an option, there will be open source methods to do what they need.

The HA page with open source options we put up will be getting more detail over time: https://influxdata.com/high-availability/

We're comitted to continued innovation in the open source InfluxDB and see this as one of the avenues for ensuring that we can continue those open source contributions.

infinotize• 10 years ago

I agree, I've invested a bit of time building a monitoring stack with influx, and one of the main reasons I chose it was development velocity vs some of the competition. And from a while back, clustering was supposed to be a top-level issue, coming just around the corner. At this point I'm going to have to bite the bullet and go with a more cumbersome but mature solution, like opentsdb.

m1keil• 10 years ago

Same situation here. This is beyond "disappointment". Total bait.

ericb5• 10 years ago

Ugh, we are a startup, we invested in influxdb on faith that it would "get there", developed around it, and now we can't afford what they want to charge for the scalable version.

Maybe we can avoid clustering with some workaround, (sharding) but I feel tricked.

pauldix12• 10 years ago

I'm sorry that you feel tricked. I hope that one of the open source options handles your use case if commercial software doesn't work for you.

However, we think that the community will be better off if we can continue to contribute to the open source InfluxDB. We want to invest heavily in open and closed source software. Ultimately the open source community will benefit from an open InfluxDB

ericb5• 10 years ago

A pricing model that scaled with usage and was more inexpensive at the low-end would, in my opinion, better serve your company and customers. That way you pull more people into the product and increase the overall size of the pie.

redwood3• 10 years ago

In all seriousness how is it possible that you're running a business and can't afford to pay what is basically a very small amount for some software?

I'm guessing you also have customers and are trying to figure out how it is that you can get them to pay you money so that you yourself can create a long-term business and create better products over time. Right?

ericb5• 10 years ago

I see you've never bootstrapped a startup before.

The clustering was (accidentally) bait and switched and led us to choose this product when we would have made other decisions.

sp1982• 10 years ago

We use blueflood developed by Rackspace at Square which uses cassandra , coupled with a query layer called MQE (https://github.com/square/metrics). While it's actively developed, I definitely recommend taking a look if you are interested in highly scalable metrics system with decent strategy for rollups. (100k+ metrics/sec).

pauldix12• 10 years ago

The reason we think InfluxDB is compelling as a standalone server is that you can get 300k metrics/sec on a single server. The scale is there for many use cases in the current release

tlipcon• 10 years ago

Hopefully this isn't too "pitch"-y, but: if you're looking for a database that's good at time series, will always be open source, and does support scale-out and HA, you might be interested in Apache Kudu (incubating).

Feel free to drop by our Slack (http://getkudu-slack.herokuapp.com ) if you have any questions.

rodionos• 10 years ago

Axibase Time Series Database is free for pseudo-cluster installations and ships with SQL and visualization included [0].

We also have a fully functional Grafana driver in case the customer prefers it over programmable visualization that we ship.

By programmable visualization I mean a way of building dashboards using toml-flavored configuration [1] and templating language, as opposed to manual design.

[0] https://axibase.com/products/axibase-time-series-database/ [1] https://apps.axibase.com/chartlab/2ef08f32

fangjin• 10 years ago

Druid (http://druid.io/druid-powered.html) is another option for similar workloads. Druid is a community-led open source data store used by many technology companies at very large scale. Comes with multiple visualization/open source applications, SQL interfaces, Grafana extensions, and a community to help with issues.

siscia• 10 years ago

An interesting way to monetize software could be to have people pay to access official docker images.

You want to pull the latest images an unlimited number of times ? Sure, it is just XXX$/month

Nothing will stop people to build open source containers of the same product, but those won't be "official" and there won't be any update guarantees...

terom• 10 years ago

Congrats on the decision. Standalone InfluxDB can be scaled up just fine to meet most usecases, and it's better to have the long-term project sustainability that a hosted/enterprise offering can bring to the standalone offering.

InfluxDB 0.9 still had plenty of bugs, and I'd rather see a high-quality standalone server than any not-quite-there-yet clustered version.

pauldix12• 10 years ago

Thanks, we do feel that this will lead to a better overall InfluxDB in both standalone and clustered.

The current version is siginficantly better than 0.9 and it's only getting better

bogomipz4• 10 years ago

Agreed I would have like to have seen these bugs ironed out before the whole splashy rebranding to Influx Data or whatever they are calling themselves now and their business pivot to a "data platform." I think the timing is unfortunate.

reubensutton• 10 years ago

I wonder if clustering will be available in the cloud service?

We use a single server on Influx's cloud service for some internal metrics and it has been running without any interruptions for >6mo.

pauldix12• 10 years ago

It will be available in the cloud service in the next few months. But if you want to continue running single server that's ok too

kev0096• 10 years ago

"less than ½ of a percent are running active clusters." because it doesn't work. They have been TheatricDB to me for a long time, this is just another nail in the coffin.

spotman• 10 years ago

... "customers eventually drop support as their infrastructures mature and they look to reduce operating costs"

This is part of the game called software. In some worlds this is actually the goal; that the software works so well and is so reliable that your customers eventually don't all need to keep paying you.

So you find new customers and add new, never advertised before features and enterprise clients with SLA support contracts, etc.

It's very understandable that it's hard to monetize, but giving people the impression clustering was going to be included long term and taking it away is not going to score you points.

Furthermore you want to get all the folks doing a startup on a budget hearing a story that works for them. Most ( especially new or young ) cofounder programmer types rarely plan to not need clustering, even if the reality is most won't need or use it. Saying this is for the big kids only is a turn off at this point in the influx story.

Wish you the best of luck

noir-york2• 10 years ago

This basically kills Influxdb for me. We're evaluating influxdb in stand-alone mode to collect limited metrics (grafana) with a view to eventually moving more and more data to Influxdb once clustering became available. $399 per month is ridiculous. Clustering is table stakes.

Lesson learnt: before deploying a new OSS, check if they have a credible plan to support the project. Otherwise skip.

akanet• 10 years ago

I mean, honestly, $400/mo is table stakes too, if you're actually using a clustered database in production for anything serious.

I think HN in general is being a bit too cynical here. This is an OSS company trying to find a sustainable way to survive. There are going to be some tricky tradeoffs. "They should just charge $20/mo for us small guys!" is not a reasonable course of action.

Regardless of whether this particular pricing plan works out or not, I think it's a good idea for programmers to support other programmers trying to make money on OSS. We are all not served very well by the general attitude of "I heard about this technology last year, it should be $0 now".

redwood3• 10 years ago

Thank you. The culture of entitlement around software boggles my mind sometimes.

noir-york2• 10 years ago

How much do you pay per month for: Apache, Nginx, Varnish, Redis, memcached, compilers, interpreters, Nodejs, Cassandra, Linux, Postgresql?

We used to pay for this stuff - remember Sun? Things change.

I suspect what happened with InfluxData is that being venture funded, they see that the funding environment has changed and are racing towards revenue/cash-flow positive. And clustering as a premium feature is, apparently, the answer.

If clustering had been pegged as a premium feature up front then neither I, nor I suspect would anyone else, have a problem. But it wasn't, hence the sour taste.

gtirloni• 10 years ago

People will find increasingly clever ways to work around the lack of clustering, as they always did. This could mean only the top-tier users will be paying, which must be a very small portion of the pie.

Even though this announcement is sold as something that will empower InfluxData to add even more cool features to the open source version, I'm actually worried that it won't be around as an open source product for very long. Let me explain.

InfluxDB is a very fine product but I don't think it has all the momentum it needs to keep going in the long term yet. That means it's still building that critical mass of supporters in an open source ecosystem. With this announcement, InfluxData is eroding the trust that small to medium-sized users had in it so the momentum slows down. If InfluxDB was a no-brainer for anyone starting a time-series project, it is not anymore. All the other alternatives need to be carefully analyzed and, even if InfluxDB is chosen, there is that voice in the back of your head saying you'll be in trouble if you exceed a single node's capacity. Eventually people will jump to the next TSDB solution that offers clustering as soon as it becomes avaiable. Then where are all those paying customers going? It'll then be easier and cheaper for InfluxDB to become a proprietary software company.

While there are some comparisons being made to what Nginx does with its Nginx Plus offering, I think it's the opposite situation. IIRC, since the early days, Nginx has offered a paid product with more features. And recently they have started to add those features to the open source version, so people got happier. InfluxDB has always promised clustering (it was a major selling point, go watch any presentation about it from 2 years ago), shipped the code (even if it's half-working as of now) and now announced it's removing the functionality. Suddenly any InfluxDB node you have (even if it's running just fine without clustering) looks like a lemon.

It's all such bad PR. And it's 2016, haven't hundreds of OSS companies been through this already? Oh well.

hoov• 10 years ago

I'm pretty damn angry right now, but I'm sure that in a few days I'll get over it.

I run the tech side of a decent sized startup right now. Having joined ~4 years into the venture, one of my immediate concerns was the lack of visibility into how our system works. Keeping tabs on the performance of several hundred ETL pipelines is not super easy. I decided to double down on InfluxDB, changing the road-map of my infrastructure team.

Then, InfluxData introduced the TICK stack. I can get monitoring and alerting as well? Let's double down again!

I bought hardware, set up a cluster (painful), we filed bug reports, learned what sorts of queries not to run, and we were in good shape. I purchased tickets for a training session and a flight (out of pocket; we don't have travel budget). And then I saw the blog post last night.

All along, I knew that the promise of all of this great technology for free was too good to be true. There was always a little voice in my head asking me about how they actually made money -- I knew that a paid offering + professional services was not sustainable.

I'm angry, but we'll be fine. We're going to break apart our cluster and shard. We've already got code written to do a backup/restore that actually works (slowly) from one cluster to another. If we hit the point where sharding doesn't work, then it'll also probably be financially viable to pay the $399/month (on top of server costs). I'll still go to the training, but I'm not sure what I'll get out of it. The reason why I was going to the training was the section on "Cluster Administration".

The only thing that I'm raw about is the lack of apology. You have to make money, and you had to do a bait-and-switch as a result. I totally understand, but the tone of that blog post was too defensive and unapologetic.

shaneduan• 10 years ago

Hello, I was directed here through the mentioning of blueflood. I hope it is ok to share some product information. Please let me know if there is a more appropriate way to do it. My only goal is to offer solution to solve the pain we run into frequently ourselves.

Rackspace Metrics (the product team behind blueflood) is currently doing many times more than 300 metrics per second. Our goal of this year is to reach to a new scale from where we are at, driven from the needs of big internal adoption this year. So blueflood project would be live and kicking.

Here is the most recent update on Rackspace Metrics product. All the functional work are through blueflood: http://bit.ly/rax-metrics-mvp

Also, there is a hidden gem in monitoring unknown for most of the startups. My other product, Rackspace Monitoring, is a SaaS product designed to assist Rackspace business instead of making big bucks. If you have a Rackspace cloud account, which requires minimum $50 service charge, you will be able to use Rackspace Monitoring for free. You can use it to monitor any other servers you host anywhere. This means that if your monitoring budget is already above $50 (all cost considered), you really should look into Rackspace Monitoring.

To learn about Rackspace Monitoring, please go here: https://www.rackspace.com/cloud/monitoring/

Please also feel free to reach out to me. We are located at the same location of Rackspace Startup Program in San Francisco. We talk to startups all the time and always enjoy the learning.

bussetta• 10 years ago

it appears that paul dix deleted his answer in this stackoverflow question http://web.archive.org/web/20150416175827/http://stackoverfl...

icbm504• 10 years ago

It seems like there are 2 OSS models other there: 1) community supported; 2) company led.

Most of the "community supported" projects that I have used are libraries. Since shifting from developer to the devops team, nearly everything that I rely on is "company led": saltstack, docker, grafana, influxdb, ELK, ngnix, sensu, rabbitmq, etc.

That being said, we are now re-evaluating the use of influxdb.

Update on InfluxDB Clustering, High Availability and Monetization

Related Stories