paulasmuth

If a beginner wanted to do anything "cool-looking" today, this same beginner programmer would need to set up render to texture on an OpenGL surface. Yuck! And so they lose interest, believing programming is the mysterious and complex domain of gods.

Or just use the web platform: JavaScript, HTML5 Canvas API, WebGL, all of that. I think it's pretty close in terms of getting a pretty (motivating) result quickly without much prior experience.

(Unrelated, but I can also remember writing the first lines of code of my life in qbasic on a dos machine in ~1998; the blue screen definitely invokes childhood memories!)

> Well, yeah. That's 28,108.80 a month

If you can't (or don't want to!) afford that kind of money for data analytics, please consider giving the FOSS alternative EventQL [0] a try some time.

It's super simple to set up and tries to be efficient on commodity hardware, so you can run large clusters (>100TB scale) for a couple hundred dollars a month.

[0] https://eventql.io/

DISC: I'm one of the EventQL authors

> For example in germany FF is still the widest used desktop browser.

Do you have any references to back that up? I'm asking because I suspect your assumption is completely wrong.

(Based on a bunch of analytics data that I have access to, which might not be representative but still contains some very large german web properties, chrome has more than twice the market share compared to FF in germany).

It's not an entirely new art form. The demoscene [0] has been going strong far longer than most of us have even been coding. So apparently some people _will_ execute interactive art.

Also, you can still listen to their music on soundcloud [1] if you don't want to review the code. It's open source after all, which can't be said for most demos...

[0] https://en.wikipedia.org/wiki/Demoscene -- Check this video for a neat example: https://www.youtube.com/watch?v=6AUpZNq2vSQ

[1] https://soundcloud.com/nemesis-fixx/

https://en.wikipedia.org/wiki/Demoscene

BTW I do actually enjoy the linked music (allthough I have to admit I'm listening on soundcloud)!

Speaking for myself: Berlin has a special kind of poor-is-sexy culture and this would not be too unusual. Lived there for 4yrs, most of them without a smartphone (had a 20eur burner phone for communication and to receive monitoring alerts). Had an absolute blast and didn't feel like I was missing a thing.

> So it's a completely different class of application than splunk or elasticsearch

Sure it takes a somewhat different approach (i.e. it requires an explicit schema), but for the use case discussed in this thread it _is_ completely relevant and a comparable open-source/on-premise alternative which parent was asking about.

> one that you have a commercial interest in

Yes, I'm involved in the EventQL project but I thought that it was obvious from the way I phrased my posting. Usually I always include a disclaimer to prevent misunderstandings but I didn't consider it necessary in this case.

> Please don't spam HN.

I don't think pitching a (relevant) startup is against the rules or the spirit of a startup forum (https://news.ycombinator.com/newsguidelines.html)

I'm still surprised that there's no startups offering comparable (particularly on-prem) products

Please consider giving EventQL [0] a try some time! It's completely open-source and self-hostable. Still a new project though, just released this summer and still in beta.

[0] https://eventql.io/

Yes, but this is only a solution if your library is only targeting windows.

If you want users of any standards-compliant C++ implementation to be able to use your library today, you'll still have to go with c-abi symbols or ship the sources. All other worakrounds are vendor-specific and not part of the standard.

[Of course even objects containing only C symbols are not portable across platforms either, but at least the C ABI/calling convention is more or less strictly defined for any given target platform. Assuming no other platform-specific stuff like glibc is used]

Thanks. I had read that article some time back :)

Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers]

EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!

Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).

Actually, you can even have incompatibilities with the same compiler and different compile flags. So to reliably build a "semi-portable" c++ object one would have to ensure that all objects are compiled with the exact same (i.e. same version of the compiler/same compiler source code) and with the exact same flags. I'm sure there are some compiler vendors that offer ABI backwards-compatibility but it's not part of the C++ language per-se.

See this article by Herb Sutter on the topic: https://isocpp.org/files/papers/n4028.pdf

That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes.

Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).

Except for inlineable functions and templates in C++—which become a greater and greater fraction of code the more "modern" your C++ gets.

The C++ ABI is currently not portable anyway. So the concern about templates (or any C++ features) forcing you to put the implementation into the header file would not apply in the scenario dllthomas was referring to:

If you want to distribute your library as a binary object and a separate source-form interface/header today, you'll have to use a (wrapper) C API. Regardless of how "modern" the C++ code is.

Of course, it would be nice to have portable C++ objects some time in the future which would change things... :)

This confused me, too.

I think what they meant with "1000 nodes" is that the dataset they're using for the benchmark is synthetic monitoring data (where the thing being monitored are servers).

And the way they generated the synthetic data set is by having 1000 imaginative servers produce one sample per second, (i.e. have a script that writes out 1000 * duration_in_sec fake samples -- I believe this is the code that does it https://github.com/influxdata/influxdb-comparisons/tree/mast...)

Just so sad that Vertica is proprietary so we can't see how they did it! ;)

On a serious note: Please check out EventQL [0] some time. It's very similar to Vertica in some ways and completely open-source. It's a new project (beta) and not nearly as mature as vertica yet though (still a long way to go).

[0] https://eventql.io/

The point of the comparison is that InfluxDB gives you a ton of functionality out of the box and has great performance. [...] if you want to do time series on Cassandra, you're going to write a bunch of the code yourself.

Fair enough. I'm sure InfluxDB is very good/fast at timeseries data (allthough I have to admit to not actually having tried it out so far). Still, if that was your point, consider removing these statements from the blog.

InfluxDB outperformed Cassandra by 4.5x when it came to data ingestion.

InfluxDB outperformed Cassandra by delivering 10.8x better compression.

InfluxDB outperformed Cassandra by delivering up to 168x better query performance.

I think it would help make the point and not put the reader in a defensive position (when the statements are clearly not based on a fair comparison of the two products and will not hold under most conditions). Just my two cents.

Yes, the delta encoding scheme you described (and other fancy coding schemes such as bitpacking, varints, RLE or a combination thereof) are frequently employed in columnar storage formats and databases. Columnar storage is basically a generalization that allows one to apply these optimizations to all kinds of data (not just timeseries). One popular open-source implementation of columnar storage that I am not affiliated with is https://parquet.apache.org/.

(On the other hand, columnar storage also has a bunch of tradeoffs/downsides so it's not a superior choice for every db product.)

My point about no "superior compression technology here" was specific to the linked benchmark. I.e. the lack of this potential optimization in cassandra does not appear to be the reason for the space blowup in the benchmark, but rather that they're duplicating the series ID for each sample.

The linked article is an obviously bullshit benchmark that makes influxdb look good and cassandra look bad (by, surprise, the influxdb folks).

I'm far from a cassandra fanboy, but this really is just dishonest marketing. Not sure if that will work if your product is open source and the target audience are developers.

Some thoughts:

- The reason why cassandra uses so much more space to store the same data is that they've set up the cassandra table schema in such a way that cassandra needs to write the series ID string for each sample (while influxdb only needs to write the values). You easily get a 10-100x blowup just from that. There is no superior "compression" technology here but just an apples-to-oranges comparison.

- Then, comparing the queries is even worse, because they are testing a kind of query (aggregation) that cassandra does not support. To still get a benchmark where they're much faster, they just wrote some code that retrieves all the data from cassandra into a process and then executes the query within their own process. If anything, they're benchmarking one query tool they've written against another one of their own tools.

- Also, if I didn't miss anythin, the article doesn't say on what kind of cluster they actually ran this on or even if they ran both tests on the same hardware. There definitely are cassandra clusters handling more than 100k writes/sec in production right now. So I guess they picked a peculiar configuration in which they outperform cassandra in terms of write ops (given a good distribution of keys, cassandra is more or less linearly scalable in this dimension)

- A better target to benchmark against would probably be http://opentsdb.net/ or http://prometheus.io/ - both seem to have somewhat similar semantics to InfluxDB (which cassandra and elasticsearch do not)

DISC: I also work on a distributed database product (https://eventql.io) but it's neither a direct competitor to Cassandra nor InfluxDB nor any of the other products I've mentioned. I hope the comment doesn't come across as too harsh. The article raised some very big (and harsh) claims so I think it's fair to respond in tone.

It seems to solve a different usecase. LLVM is (by their own documentation) not really intended for ad-hoc JIT compilation for queries. It doesn't put a lot of priority on making the compilation fast (but rather makes the produced code fast).

On top of that, LLVM is a massive dependency.

I only looked at NativeJIT a few minutes but it seems to solve both of these issues. It's pretty small in terms of API surface // total amount of code and is explicitly designed for a usecase where compilation happens as part of a user-issued query (that needs to be fast).

What I find amazing about NativeJIT is how small the codebase seems to be -- under 10k SLOC. Impressive!

Will seriously consider using this to speed up expression execution in EventQL [0] (we shied away from llvm so far because it's such a massive dependency).

[0] https://eventql.io/

I can't speak for big query, but for our open-source BigQuery alternative EventQL [0].

On the upside, massively parallel SQL databases allow you to express what probably took facebook a lot of code in a single SQL query.

On the downside, with any database that supports streaming inserts and dynamic resharding, you will have to eventually write each entry more than once. The more manual approach with spark prevents this.

[0] https://eventql.io

All of those points sound fantastic and zig is one of the most interesting things I've seen on HN in a while.

Still I can't help but ask one question: Your points 2 and 3 (build tooling and package management) seem like they would have the biggest, immediate impact for most current c/c++ users.

Did you ever consider separating the build tooling and dependency management effort from the core language?

I, for one, would be extremely interested in sane build tooling and dependency management for c/c++ (and what you describe sounds great). I would give it a try in no time. However, I'd be much more hesitant to consider a new language for a serious project. Even if the language is tip top, I would still be concerned if it is likely too be around and still relevant in X years. I think build tooling has less of a chicken-and-egg problem to adoption because you could always switch to another tool later. And once somebody uses your build system you're already half the way there to using the full thing.

None of these numbers add up.

If 30 million hours were watched with an average bandwidth of 2Mbps, the total data volume should be 25,749TB (or close to 26PB) and not 400TB.

Also, if the peak bandwidth was 600Gpbs, that peak could not have lasted very long. With that much bandwidth the claimed 400TB of total transferred data would be used up in roughly 90 minutes.

So either the total data transferred or the number of hours watched figure is off by a factor of at least 60x.

[ I figured "400Tb" was supposed to mean 400 terrabytes. If it's actually 400 terrabits, the numbers are off by one magnitude more ]

If you care about not having your data locked-in to Google's ecosystem or think BigQuery is too expensive, please consider to give the open-source EventQL [0] alternative a try.

[0] https://eventql.io

Excellent idea! Are there any benchmarks of how the changed sysctl values affect performance?

That's not a fair comparison though. I only quickly read the postgres benchmark you linked, but it looks like you stored the data on disk (!) on a machine with only 16GB of ram for the postgres benchmark, while storing the data completely in 96GB of very fast RAM for the mapd benchmark.

Unless I misread the post it's just comparing apples and oranges. If anything, the two benchmarks show that one piece of hardware (RAM) is orders of magnitude faster than the other one (SSDs). My previous comment was specifically about MySQL running on a machine where the whole dataset fits into memory.

Of course, I'm sure mapd is faster than mysql/postgres for some usecases. But the benchmark doesn't prove that in a fair comparison.

EDIT: Maybe I should add a disclaimer: I'm the founder of another open-source database product that could be considered a competitor to MapD.

256GB is 256 billion bytes. So you can only fit over a hundred billion rows into 256GB if you have zero overhead and every row is smaller than roughly two bytes... ;) [so you couldn't even store a single proper integer in each row in your example]

Also, querying a static dataset that you can fit completely into RAM is not exactly "intractable to handle in real-time" without a GPU. You can do that just fine using the CPU right now.

Case in point: Even MySQL can handle 256gb of rows in memory with ease if you give it that much ram... Essentially, any single-machine database (gpu or not) can only process the same datasets that traditional single-machine databases (like MySQL) can already handle, albeit maybe a bit faster.

I think a good example for a "newsql" usecase that does _not_ fit into classical solutions like MySQL or Postgres would be web tracking (clickstream) data: Even a medium sized web property (say alexa top 100 in germany or france) will generate dozens of gigabytes of tracking data per hour. over the period of months, this adds up to hundreds of terrabytes of data. There is no way to load that much data in one piece into any of the classical databases.

EDIT: Maybe I should add a disclaimer: I'm the founder of another open-source database product that could be considered a competitor to MapD.

To me the problem seems more that on a GPU you can only execute fast queries on fairly small datasets. I would assume that the space of problems that still fit into a few gigs of GPU RAM but need to be answered rapidly with a lot of parallelism is fairly small.

In my anecdotal sample, most of the users of hadoop/bigquery/etc seem to use it less because of raw query speed but more because the datasets are simply to large for any classical solution (i.e. much larger than fits into RAM on a single machine)

Also check out this blog about internals of massively parallel databases such as MapD (it talks about _why_ they are so fast) [0]

[0] https://news.ycombinator.com/item?id=12291767