ewencp

The downloads page is only for official releases, RCs are not distributed the same way and are not permanent unless promoted to an official release.

See the mailing list thread for 2.8.0-RC0 for where to find the bits if you want to test https://lists.apache.org/thread.html/r16894a11aec73abac521ff... and the project site has some "contact" info for mailing lists where these things are announced and advertised (including releases, Kafka Improvement Proposals, and more) https://kafka.apache.org/contact

terraform: 2014 ansible: 2012 openstack: 2010 cloudera: 2008 hortonworks: 2011 emr: 2011

(those were quickly googled, apologies if i got any dates wrong)

timing is important. cloudera had good, early timing and looked promising because of it. you are right that EMR definitely hurt all the other hadoop vendors, though i think people overestimate how comfortable big enterprises are with moving to public cloud. way more comfortable today; back then everything was a lot less certain. cloudera's name is unfortunate given they never got anything cloud-based successful, maybe they'd be in a better place now if they had.

but some of the key technologies you suggest as better options came 4-6 years later. that's 4-6 years of providing value, gaining traction, and building a committed customer base. 4-6 years is a long time, and even with how slow many enterprise projects run, more than enough to get entrenched, build tooling that makes a bunch of stuff easier, build mindshare, etc.

Have I missed out on some cool stuff?

stuff that makes companies money doesn't always look cool.

Regarding your first issue, this is very much a matter of defaults. I can't be sure of your exact pipeline and connectors, but if, for example, you were using the JDBC connector, it has included support for at least prefixing names since the original version, effectively supporting the namespacing you require https://docs.confluent.io/current/connect/connect-jdbc/docs/.... I agree this might not be as ideal as namespacing directly at the Kafka layer for some users. The addition of single message transforms to arbitrarily modify the topic names (based on the existing topic name or really any data in the record or any info in the transformation config) gives a lot more flexibility as of Kafka 0.10.2. On the Hadoop/Hive side, I think there may still be that limitation; transformations effectively remove it since you can arbitrarily adjust the topic the sink connector sees, but this probably isn't an obvious solution. Also, we really would prefer to avoid any coding required when using Connect. It's a difficult tradeoff between standardization (same configs everywhere), usability (minimize configs the user has to set), and simplicity+immediate usability (transformations came later and introduce configuration complexity). I (and other Kafka contributors) are certainly welcome to thoughts on how to make this all simpler; I think most software, especially open source software, errs too heavily on towards configurability, but clearly in your case you found things not configurable enough.

re: the point about backpressure, there are plenty of cases where you don't want backpressure. If you want the thing that's producing data to keep humming along even if some downstream app (lets say Connect dumping the data into HDFS for some downstream batch analytics), you don't want to see backpressure. In Kafka you should just define your retention period to be long enough to cover any slowness/lag in consumer applications -- it's pretty fundamental to its design and use cases that it doesn't have explicit backpressure from consumers back to producers. (You do get backpressure from a single broker back to the producer via the TCP connection, but I assume you meant from consumer back to producer.)

I should also mention that system/integration tests of the new functionality was a requirement for releasing it. For example, here's one of the tests https://github.com/apache/kafka/blob/trunk/tests/kafkatest/t...

It's not Jepsen, but we actually do a fair amount of system and integration testing, some of which does things like kill nodes (randomly, the controller, etc) and validates data is delivered correctly. There is some ongoing work to add other fault injection tests: https://issues.apache.org/jira/browse/KAFKA-5476

This has already caught quite a few bugs, including one that led to a change to the replication protocol which is included in this 0.11.0.0 release. See the community KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+...

One cool thing that happened recently with these tests is that they were modified to make the client implementation pluggable: https://github.com/apache/kafka/pull/2048 Confluent uses this functionality to test all of its clients (librdkafka, confluent-kafka-python, confluent-kafka-go, confluent-kafka-dotnet) in addition to the Java clients. This not only makes us confident of these clients from their first release, but has also found dozens of bugs in both the clients and the broker implementation itself. Getting automated testing across many clients has really stepped up the quality and robustness of both existing and new features.

If you're interested, the tests themselves are here: https://github.com/apache/kafka/tree/trunk/tests

Debezium implements Kafka Connectors, so the serialization is pluggable. Debezium can work with thrift, you just need a thrift converter for Kafka Connect.

It's not particularly efficient, but probably the easiest way to draw Voronoi diagrams on the GPU doesn't require a shader at all. Instead, use an orthographic projection and draw cones for each vertex with the cone's apex at the vertex and its axis oriented into the screen. Then the GPU's z-buffer takes care of choosing which vertex is closes to the given pixel.

For apt repositories, you might be interested in http://www.aptly.info/, especially if you want to host it on S3 as it integrates very well. As others have mentioned reprepo isn't that tough to use either, but moving it to S3 instead of somewhere else basically amounted to using an S3 URI. It also has some other features that might be handy, e.g. versioning/snapshots and serving your repos locally for testing.

In any case, I find the process of actually creating the packages far more arduous than setting up the repo...

In particular, http://kafka.apache.org/documentation.html#intro_consumers addresses the concept of consumer groups and what ordering is guaranteed. One thing that might be worth noting for the grandparent is that Kafka consumers have an offset commit API that gives some control over how failures are handled. If a consumer dies before committing an offset but after reading data from the broker, a fresh consumer that joins the consumer group can see the same data once the system determines the original has died; that ensures all data will be processed, even in the event of consumer failures.

Kinesis provides the same ordering guarantees. They use different terminology (Kafka topics == Kinesis streams; Kafka partitions == Kinesis shards) but have the same system interface. The details of the APIs used for consumption differ, but they provide the same basic functionality of Kafka's "consumer groups".

Agreed! It's not Docker, but I'm working on getting a decent Vagrant setup included with Kafka: https://issues.apache.org/jira/browse/KAFKA-1173 That supports pulling up a full cluster locally in VirtualBox or in EC2. Just a first cut, but it already makes testing a lot easier for me.

But in it's current state, that patch is a starting point that is really intended more for Kafka developers than for Kafka users. I really like what the Mesosphere folks have done -- great variety of OSes and cloud platforms, plus they do all the heavy lifting of bringing the cluster up for you.

Along similar lines, all of the English Wikipedia is < 10GB, and about 45GB uncompressed: http://en.wikipedia.org/wiki/Wikipedia:Database_download#Eng.... That omits all the history (just the current pages), but still surprising to me how small it seems now.

The golang folks did a nice write-up on profiling Go code: http://blog.golang.org/profiling-go-programs and it covers both CPU and heap profiling. They've incorporated Google's perftools directly with Go which makes the whole process very pleasant compared to what you'd need to do for many other languages.

This is correct, here are the physical measurements:

Nexus 4: 133.9 x 68.7 x 9.1 mm

Nexus 5: 137.84 x 69.17 x 8.59 mm

That said, I still find it frustrating -- the Nexus 4 screen is already too large for me to comfortably reach all parts with one hand.

Many years later, similar functionality made it into gdb: http://sourceware.org/gdb/wiki/ReverseDebug

Given the huge variation in (mis)calibrated settings on top of the huge variation of displays themselves, you'd probably be fine running f.lux. In fact, I would think you would want a more flexible tool so you can actually test what different users might be seeing given different display configurations.

Thanks for trying it out. Sounds like a bug in the collection editor. I want to improve that process to handle cases like yours -- if you don't find the feed, make it easy to look it up and add it -- but I wasn't sure that was top priority since a lot of people end up using feeds that are already in the system.

Ah, sorry, I was using it in the locality sense (as in, coherent access). Admittedly less common, but not uncommon, in algorithms; downright confusing if you're in an architecture frame of mind.

I'm just learning Go, I'm certainly no evangelist. One huge difference for me, however, doesn't have to do with the language. It's deployment. It's really nice for deployment to be scp rather than a bunch of package installs, making sure your classpaths are right, etc.

That said, one of the language features that drew me in is structural typing. I'd yet to see a well supported statically typed language that used structural typing, and it seemed like it would make software maintenance a lot easier. Of course, tools are often a better solution than language features, so maybe that's a bad reason to use a language.

For the vast majority of cases, I would expect panic/recover to work for iteration over in-memory data structures. For something like a database iterator where the possible errors are more varied and likely, I agree an explicit iterator object is better. I probably should have defined the scope of problems I was thinking about better.

To build on thrownaway2424's response, the point in the article was to demonstrate the pattern for building the iterator. Very simply structured data helps do that. In reality you'd probably be iterating over a more complicated data structure, usually where the iteration requires knowledge of the data structure that should be considered private to the implementation.

Fair enough. My day-to-day coding with Go right now involves large data sets where materializing something like that would be expensive. I'll admit my view of the issue is strongly biased by my very real need for iterators or something equivalent that can avoid that cost!

I mentioned this pitfall, but didn't show how to solve it. I haven't seen a complete example yet and didn't take the time to work out the details yet. I'm less concerned about complexity in the iterator implementation, but handling this correctly also seemed to require changing the caller code since it needs to send something on the fail channel. Then the onus is on the caller to remember to do so, which is just as bad as having to remember not to call break.

Of course, whether this is a real issue or not also depends on whether you're writing internal or external APIs. There are lots of other ways to generate uncollectible garbage too...

Testing performance by iterating over an array can be dangerous because all the data is conveniently grouped together such that the processor will have the next data it needs in cache once it's loaded the first item (most of the time; when it crosses cache lines it'll cause another miss, then a bunch more hits). This can lead to huge performance boosts, but rarely reflects how data is laid out in memory unless you're very lucky.

The change made was to still iterate over a slice (i.e. an array), but the array now contains pointers to structs, which are allocated separately. This should scatter them across memory such that each step of the iteration needs to choice a different pointer to a different chunk of memory. This is a benchmarking issue which has bitten me in the past, so I made at least a feeble attempt to account for it.

However, I actually don't know enough about Go's memory allocation to really make sure I could force the conditions I wanted, so it's still possible all the data was lined up nicely, but just spread a bit farther apart. (In fact, this may even be likely since the allocation approach was very simple. Trying to do a lot of allocations and deallocations and filling in the data over a longer period would probably do a better job of addressing this concern.)

I'll buy that -- it's sort of mentioned in the post as an object-based wrapper around the closure version, except I listed the functions as HasNext() and Next(). More boilerplate for the iterator implementer, but I agree, it reads better for the caller.

You need them because the examples are intentionally simplified. Even a data structure as simple as a dictionary has a useful variety of iterators: over keys, over values, and in some languages, over (key,value) tuples. Explicit iterators are really only noise for the "natural" iterator, i.e. in order iteration over an array.

I agree that having it hidden by a language feature is nice, but somewhere there is an iterator -- the class that supports that iteration must at least specify how it's accomplished. I would prefer if there were an easy way to integrate with the range keyword in Go.

EDIT: Adding to reply to some added comments. The close() one in particular really threw me. It makes sense in terms of how channels work, but is really unfortunate. When I first saw the channel-based approach, I thought it was a neat solution. But then I found all the problems with it (including the best example I could find of it online missing that close(), which is why I pointed it out!).

I think Go is close to getting it right if they would do what I mention at the end of the article: provide a way to hook into the range keyword.

Should probably qualify: MBP from 3-4 years ago... But I don't expect it should suck this bad this quickly....

Not in my experience -- I have the same issues on my MBP, though I use it maybe 1/10th the time, so I'm less certain of that impression.

I honestly can't give hard numbers, but I really did switch back to FF because of the memory usage. My ancient-by-HN-standards but not-too-ancient-by-normal-standards desktop with 4GB RAM (Ubuntu 10.04 if that's relevant) was starting to thrash like crazy with a moderate number of tabs (maybe 6-12 on average), but most importantly, with a few open for long periods of time. Firefox seemed to really clean up their act wrt memory, where Chromium seemed to grow without bound and keep a lot of it in the working set, doubling the pain.

Firefox isn't perfect, but it seems to be more of a problem with Flash than the browser (clearing out my Rdio tab has the most effect). And measuring memory usage through top isn't perfect, but FF does seem to use significantly less RES even after long sessions.

So, anecdotal evidence, but I was using Chromium as my regular browser for years and ended up switching back to FF because I found the performance on my reasonable-but-aging machine to be far superior to Chromes under regular, long-term usage.

What you're quoting refers to the scaling of the axis after all the values are computed. This affects the numbers on the Y-axis, but not the shape of the graph.

The issue wting was raising is the definition of search volume. If search volume means "raw total number of searches", then the graph might support the OP's conclusion (if, e.g., we assume the population remained the same and that the same set of people aren't increasing the number of times they search for 'software consulting'). If, on the other hand, it means "fraction of all searches containing the term", then it might not because, as wting points out, the fraction of searches for 'software consulting' may go down due to a flood of searches for, say, 'honey boo boo'.

At least on the Twitter web site, they already have the unwrapped URLs -- see the data-expanded-url and data-ultimate-url attributes on links (and, of course, at least one unwrapping is used for displaying the link).

They probably don't want to completely undo other shorteners, however, because these services aren't only used for shortening. I absolutely believe it's beneficial for them to have multiple levels of indirection so others can gather stats, I just don't like it as a user who is consuming the links. If the amount of indirection, delay in loading pages, and brittleness of links gets bad enough, maybe it eventually will make sense for them to try harder to avoid it.

But they could, in some cases, avoid increasing the amount of indirection and still gather the same stats. For example, on their site, since they open links in a new tab anyway, they could make that link direct and capture the click event with a separate AJAX request in the original Twitter tab.

If you focus only on page load latency, other sites/services besides URL shorteners are making things worse. I regularly notice some small delay waiting for my click on a Google result to go through Google's servers since they (annoyingly, in my opinion) swap out the target URL for a wrapped version at the last minute. Of course, Google has other good reasons for doing this as well -- stats and removing referral information immediately come to mind.

I also didn't show that this negatively impacts user experience. It could be that t.co, on average, requires so little overhead that the extra level of indirection doesn't matter. But I think it's worth looking into, and if it does have a negative effect, figure out ways to mitigate these effects.