HN user
Programming, scalability and reliability. http://erik.gorset.no/
One of the advantages of of protobuf I never see anyone highlight is how neat and well-designed the wireformat is, in terms of backward/forward compatibility and lowlevel stuff you can do with it. Very useful when building big and demanding systems over time.
For high performance and critical stuff, SBE is much more suitable, but it doesn't have as good of a schema evolution story as protobuf.
This brings back memories debugging an azul zing bug where an effectively final optimization ended up doing the wrong thing with zstd-jni. It was painful enough that I couldn’t convince the team to enable the optimization again for years after it was fixed.
I agree. It’s the enshittification of the internet. Luckily we still have infrastructure providers with more sensible offerings. We don’t have to use aws, gcp, etc.
I've used bazel for about 10 years, all in small orgs. Right now, single digit team.
I don't think there exists a better solution for a project mixing Java and C/C++ than Bazel. The new module system for bazel has matured a lot. As an example, it's trivial to add boringssl and access it from Java.
That’s the default in the monorepos I’ve worked on.
When a third party dep is broken or needs a workaround, just include a patch in the build (or fork). Then those patches can be upstreamed asynchronously without slowing down development.
I enjoyed both talks a lot.
I really appreciate the work done to evolve Java. They seem to be able to both fix earlier mistakes (and there were many!) and new ones from happening. While I see other languages struggling with the weight of complexity added over time, Java seems to get simpler and easier even with significant updates to the language spec.
I think the «best practices» found in Java enterprise has meant that a lot of people think all Java has to look like that.
High performance Java is awesome and can be competitive with C++ and Rust for non trivial systems, thanks to much better development cycle, debugging and high quality libraries.
Most the benefits is the JVM, so Kotlin has those too, but I don’t feel Kotlin is enough of an improvement to warrant the downsides (at least for me). But both Kotlin, Scala and Clojure are great for the Java ecosystem.
Mechanisms for getting the linux kernel out of the way is pretty decent these days, and CPUs with a lot of cores are common. That means you can isolate a bunch of cores and pin threads the way you want, and then use some kernel-bypass to access hardware directly. Communicate between cores using ring buffers.
This gives you best of both worlds - carefully designed system for the hardware with near optimal performance, and still with the ability to take advantage of the full linux kernel for management, monitoring, debugging, etc.
Isolating a core and then pinning a single thread is the way to go to get both low latency and high throughput, sacrificing efficiency.
This works fine on Linux, and common approach for trading systems where it’s fine to oversubscribe a bunch of cores for this type of stuff. The cores are mostly busy spinning and doing nothing, so it’s very inefficient in terms of actual work, but great for latency and throughput when you need it.
In a previous project we used fixedint32/64 instead of varint values in the schema for messages where performance was critical.
That left only varint used for tags. For encoding we already know the size at compile time. We also don’t have to decode them since we can match on the raw bytes (again, known at compile time).
A final optimization is to make sure the message has a fixed size and then just mutate the bytes of a template directly before shipping it off. Hard to beat.
In a newer project we just use SBE, but it lacks some of the ergonomics of protobuf.
Nice work! I love the care and discussion around low level optimizations, as such things are often ignored.
There are a lot of interesting variants of rank/select and other succinct data structures which has interesting tradeoffs. Maybe most useful when compression is critical. At least my own data structures can’t compete with the query performance you are showing, but they are great when the memory footprint matters.
The number of unique int32 values is not that great, and a full bitset would only be 512MB. Hard to bit a dense bitset in performance.
As a general purpose data structure, I would look at roaring bitmaps for this purpose, which has good trade-offs with great performance for most use-cases.
If only simple lookups are needed over a static set, then it's worth looking at minimal perfect hash functions (https://sux.di.unimi.it).
Selective testing is great also for smaller projects to help keep velocity high for merging and working with branches with many commits.
I implemented selective testing using bazel for a CI some years ago, and it was painful to get it right. When finished even bigger branches would only take seconds-to-minutes to go through the pipeline, which was a significant improvement from the ~30 minutes build when I started working with the project, even though the project size grew a lot.
Glad to see mill-build is prioritizing this feature.
I'm a big fan of this solution! It's always been annoying to perform all the ceremony involved in deploying a system with a bunch of files, with config, scripts and system written in a bunch of different languages.
In my current project I just wrote the installer and config generation as part of the main method. Gets rid of a lot of complexity, with a simpler build, and is arguably easier to maintain. Single language, single binary.
I always found the existing bazel integrations annoying. My solution has been to generate .classpath based on a set of bazel queries. Not what's officially recommended, but it has worked great for the projects I've been involved in. Easy to understand, and keeps bazel completely separate and independent from the IDE experience which has some benefits.
There's a lot of ceremony to run jdtls!
I find it easier to just install language servers using brew. Then the command can be simplified to
cmd = { 'jdtls', '-data', home_dir .. '/.local/state/jdtls/' .. project_name },
They could also have gone the commercial route and gotten Zing with their pauseless GC. It’s been around forever and they even cover Cassandra in their marketing.
Patch series comes from the linux kernel workflow, which git was developed to support. https://kernelnewbies.org/PatchSeries
In this workflow you review every commit and not just the branch diff. Each commit is crafted carefully, and a well crafter series of commits can make even very large changes a brief to review.
It takes a certain skill to do this well. As the page above says > Crafting patches is one of the core activities in contributing code to the kernel, it takes practice and thought.
This is in contrast to using git more as a distributed filesystem where you don't care particularly much about the history, and you typically squash commits before merging etc. It's simpler and easier to work this way, but you lose some of the nice attributes of the linux kernel workflow.
The use case is when you look at a branch as a series of patches.
Reviewing a clean set of commits is much easier than a branch full of mistakes and wrong paths taken.
Useful when we optimize for reviewing and good history for future maintenance. This has been important and useful when I’ve worked on big mission critical backend system, but I also understand it might not be the most important factor for a new project’s success.
I struggle to use lmdb when using resource isolation (like docker, nomad exec driver). The hidden complexities really become apparent when you build largish systems using shared infrastructure. Anything is easy if you have a dedicated machine or have dataset smaller than your memory allocation.
Mmap is so simple for the application/developer, but the price you pay is giving up clear ownership of memory and clean separation between kernel page cache and application space. The interaction between the mmu, mmap, tlb and the linux kernel IO layer is complicated. You don't want to touch that complexity at all if you care about ultra low latency.
The article rephrases the question into
"do you really want to reimplement everything the OS already does for you?"
I was asking around in my network after experience with self hosting S3 like solutions. One serious user of SeaweedFS recommended looking into min.io instead. Another serious user of min.io recommend looking into SeaweedFS instead…
There are some workarounds. In my company we have a script that generated IDE files automatically. It uses `bazel query` and `bazel build` to find all external and non native dependencies, and then generates IDE config and files for rust, java, python, etc.
Positives: your IDE experience becomes native and you don't need to interact with bazel for normal IDE work flows. Negatives: need to run the generate script anything non native changes. Also need to deal with bazel build files etc for the git workflow, obviously.
We're small company, and this method has worked great, and we have a pretty complex build with python accessing rust and java (via jni). Java accessing rust and c (via jni).
I have fond memories of my grandfather serving RC Cola to me in a small town in Norway 30 years ago. Imported to the town by a small independent shop focusing on cheaper alternatives to the big brands.
Few years ago, IKEA sold RC Cola in their restaurant in Norway, but haven’t seen it recently.
Still pops up once in again through catering services, probably because of a lower price point.
The lack of “interface” was one of the main reasons I never ended up using scheme to build production systems.
I loved playing with scheme in my university days 20 years ago, and loved how one can make up any fancy programming concept using call/cc and dynamic-wind. Except if you want to have multiple implementation of some protocol or collection. Hand roll your own object model or dynamic dispatch mechanism? Hard to make it scalable, composable and compatible with other projects.
We use gitlab and review a commit at a time and do CI runs on each commit in the MR branch. Just plain git with the fixup/interactive rebase dance. Gitlab is not optimized for this workflow, but it’s liveable.
Gitlab-runner can execute any shell commands, so it’s easy to just loop through all the commits and run bazel test.
Gitlab has improved lately and it’s now easier to review each commit in a MR, but you can sense that most of the PM attention is on the single commit/squash everything MR workflow.
Actually you have both 32 and 64 bit wire types:
- wire_type=1 64 bit: fixed64, sfixed64, double
- wire_type=5 32 bit: fixed32, sfixed32, float
Pregenerating protobuf messages you want to send and then modifying the bytes in-place before sending is going to give you a nice performance boost over "normal" protobuf serialization. It can be useful if you need to be protobuf compatible, but it's obviously better to use something like SBE - https://github.com/real-logic/simple-binary-encoding
It's obviously possible to do protobuf with zero parsing/marshalling if you stick to fixed length messages and 4/8 byte fields. Not saying that's a good idea, since there are simpler binary encodings out there when you need that type of performance.
Aside from the mmap() call, you can do all this without even allocating any memory at all.
I think this is the important point when it comes to discussing zero-copy. I've written a custom protobuf implementation for java which can do exactly that.
It's a bit tricky since protobuf supports recursive messages and java's Unsafe is not as powerful as what you can have in C++. My trade-off was to require the caller to pre-allocate messages needed before parsing the data. This works great when working with multi-gigabyte files where you want to process a large number of (possibly nested) messages, but is not as ergonomic as normal protobuf code.
It obviously doesn't come for free, as you need to do a linear scan to find those tag-values, but there are ways to speed that up too, so it becomes very fast in practice.
I'm sure Cap'n Proto and FlatBuffers are faster for some use-cases (I haven't tested), but a very important point for me is to be wire-compatible with protobuf3 and its ecosystem... and still be zero-copy/zero-alloc.
An array of sorted hashes is also nice combined with interpolation search [0] (or hinted binary search) which can give a nice speedup compared to "naive" binary search.
Nice to see more userspace implementations of wireguard.
I've been testing out wireguard's linux kernel module, but experienced too much jitter due to wireguard not respecting isolcpus.
Probably possible to make the kernel module behave, but working with userspace is so much easier. Wireguard-go seems to behave (the drop in performance is not necessarily a showstopper), although the description says that it shouldn't be used on Linux :-)
Would be nice if a high performance rust implementation can become a real alternative. A userspace implementation shouldn't need to be much slower than a kernel module if network card with kernel bypass is available, if you really need that extra performance.