mpercy

Quite literally a fender bender.

It doesn’t give any examples of open- vs closed-skills sports in the abstract…

I believe it. Denying delicious El Farolito burritos to New Jerseyans would be tantamount to a war crime.

He has the advantage of knowing what is there and they can create an arrangement to license or sell the IP to another entity… if it’s just gathering dust, why not?

How are people actually using vector databases?

The closest explanation to a use case architecture I've seen recently was https://mattboegner.com/knowledge-retrieval-architecture-for... - it basically describes doing knowledge retrieval (keyword parsing) from LLM queries, feeding that to a vector db to do similarity search to get a top K similar documents to the parsed keywords, then feeding that list that back into the LLM as potential useful documents it can reference in its response. It's neat but it seems a bit hacky. Is that really the killer app for these things?

Who is developing this product? Under what circumstances is it 30% faster than Parquet?

Cuy is guinea pig, not rat, but I hear you on the rodent flavor. Worth it once for the experience but I wouldn’t eat it a second time!

This is really neat. One piece of feedback is it begins and ends TeX expressions by saying “dollar” which is distracting. Probably best to strip the TeX syntax while retaining the expressions. Simple ones like O(1) should be understandable aurally, even if complex expressions may not be.

There is no doubt.

Most people have good health care plans through their employer that pays for most of this stuff. I paid a few hundred dollars to the hospital when my child was born and we were there for several days. There are just a lot of people who for one reason or another do not have those benefits. It’s a very multimodal experience here unfortunately.

FF as in Firefox?

Not true. Carter gave a blanket pardon to all Vietnam draft dodgers, many of whom fled the country before being convicted.

Lovely article. Allosaurus is amazing and terrifying. Very cool factoids about the bunny hands, first time hearing they were incorrectly depicted.

Since I mentioned performance, one other area that makes flexibility nontrivial is how you decide to serialize different types of messages on the wire and on disk. If you don't need extensibility, it's easy to keep things pretty efficient just by using e.g. gRPC and protobuf out of the box. If you want complete flexibility, the simplest thing to do is to give your plugin interfaces blobs to write into and you end up double-serializing everything.

Sure, here are a couple examples of things we've had to include in the log APIs:

1. Kudu uses a special "commit" record that goes into the log for crash consistency, related to storage engine buffer flushes. So we need an API to write those into the log. They don't have a term and an index, since they are a local-engine thing, so they have to be skipped when replicating data to other nodes in the case of the current node being the leader. If we were not sharing the log with the engine, we wouldn't need this.

2. Another database I'm working with requires file format information to be written at the top of every log segment, and it has to match the version of the log events following it. That info has to be communicated to the follower up-front even when the follower resumes replicating from the middle of the leader's log. So we need plugin callbacks on both sides to handle this, in terms of packing this as the leader and unpacking it as a follower into the wire protocol metadata.

Requirements like these will come up and either you hack around them by making some kind of out-of-band call (not ideal for multiple reasons) or you bake the capability into the plugin APIs and the communication protocol.

Frankly, designing generic APIs is also one of the less sexy aspects to consider because we spend so much of our time dreaming about and building all the cool distributed systems capabilities like leader elections, dynamic membership changes, flexible quorums, proxying/forwarding messages, rack/region awareness, etc etc etc. :)

The details of long-tail stuff like this is often hammered out as it comes up during implementation.

From experience working in this area, I believe there's a significant tradeoff between performance, flexibility, and time to delivery when it comes to consensus and the things it's used for, like database replication. It's like: "good, fast, or cheap, pick two".

As one of the core authors of the Apache Kudu Raft implementation <http://kudu.apache.org/> (which is written in C++) I know that we tried to design it to be a pretty standalone subsystem, but didn't try to actually provide a libraft per se. We wanted to reuse the Raft write-ahead log as the database write-ahead log (as a performance optimization) which is one reason that making the log API completely generic eluded us a little.

That said, I'm currently at Facebook helping to adapt that implementation for another database. We are trying to make it database agnostic, and we continue to find cases where we need some extra metadata from the storage engine, new kinds of callbacks, or hacks to deal with various cases that just work differently than the Kudu storage engine. It would likely take anybody several real world integrations to get the APIs right (I'm hopeful that we eventually will :)

Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.h...).

I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern.

I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus.

By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285

We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.

FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.

I have been working full-time on Kudu since its early development. As others have mentioned, Arrow and Kudu are quite different. Despite the controversial-sounding title of Daniel Abadi's article, his content was actually reasonable and his conclusion in the final paragraph of the article is worth reading. In summary, he acknowledges that in-memory and on-disk columnar formats have different goals and both have their place (Arrow being an in-memory format).

Apache Kudu is much more than a file format - it is a columnar distributed storage engine. One way to think of Kudu is as mutable Parquet, but really it's a database backend that integrates with Impala and Spark for SQL, among other systems. It's fault tolerant, manages partitioning for you, secure, and much more. For a quick introduction to Kudu you can check out this short slide deck I put together over a year ago... it's a bit dated but a good overview: https://www.slideshare.net/MichaelPercy3/intro-to-apache-kud...

For more up-to-date information, follow the Apache Kudu Blog at http://kudu.apache.org/blog/ or follow the official Apache Kudu twitter account @ApacheKudu.

I think it's because Qt has traditionally had a pretty weird dual- or tri-licensing model: https://www.qt.io/faq/#_Toc453700684 which resulted in less adoption by developers because of the viral nature of the GPL.

IIRC, it used to be that even the core Qt libs were GPL (unless you paid for a commercial license), while now most (but still not all) libs are also available under the LGPL.

GTK has always been plain LGPL 2. Nothing scary.

Run your tests against clang (as well as gcc) and then ship with gcc. Last I heard, gcc still had the speed advantage for optimized code anyway.

Fred, really interesting article! Thanks for posting it.

For those in the Bay Area, if you'd like to hear more about Kudu, I'm giving a talk about it in Palo Alto tomorrow (Wed) at the Big Data Application Meetup at 6PM: http://www.meetup.com/BigDataApps/events/227191025/

Also presenting are Julien Le Dem of Dremio on Apache Drill and James Taylor of Salesforce on Apache Phoenix. Should be a really interesting evening!

(Don't mean to hijack the thread, just thought this might be relevant to some folks. :)

Mike

Fair enough, but here's another example. Say someone is trying to move from IC to management. They might have no mgmt experience so they put "seeking engineering management position" as their objective. Is there a better way? The other traditional thing to do would be to explain this in your cover letter.

Edit: I get that you should be shopping it around and applying to various positions but it seems inevitable that a copy of your resume will get filed into countless HR databases. It seems rational to me to indicate what you're looking for on the resume.

Hrm. I'm not looking, but I like reading these things. Regarding "Objective, seriously? It's not 1992 anymore", I've always put an objective on my resume. Typically something short and sweet, like "seeking a senior software engineering position solving challenging systems problems" or whatever. Is that not useful? I wouldn't want someone to read beyond the first couple lines of my resume if they're looking for a web developer or a finance manager.

We're using libev (actually libev++ which is cross-platform, has a decent underlying C impl, and a C++ bridge API). It seems to work well and is fast and cross platform. It can be a little tricky to figure out what is going on with the C++ API sometimes due to how the author tacked it on using macros.

John does a good job of explaining the difference between software patents and traditional hardware patents to the layperson. He notes that while hardware patents are usually very specific, software patents can be so general as to cover ideas that the patent holder "had never even dreamed of". He also strongly implies that the US Innovation Act <http://en.wikipedia.org/wiki/Innovation_Act>, after being passed by the House, was killed in the Senate by lobbyists representing trial lawyers.

If you haven't seen Brendan Gregg's Linux performance tools page & slides, you should check it out. He explores a wide variety of great stuff, especially perf and ftrace (as mentioned in the article) as well as a few dozen others that he describes in lesser detail: http://www.brendangregg.com/linuxperf.html

Wow... I understand that you'd be much more productive with a large monitor and good keyboard, but to the extent that you couldn't perform in a coding interview? Give me a break...

In the past, I've used CollabEdit for "phone screens" and the whiteboard for in-person interviews. I try to ask some "coding" questions (identify a better data structure / algo for a particular scenario and then implement it in code) and some problem-solving conversational questions. Seems to work pretty well, but I'm always on the lookout for better approaches. Part of the problem is that the software field is so broad that it's hard to get a sense of a person's abilities with such a limited amount of time to ask questions.

I thought you were wrong but not so. Here's the proof that they are indeed the same company: http://pivotal.io/labs

Whoops, looks like our posts crossed on the wire. I definitely think Steve's post is worth the read - it hits home.