dave_ops

Having had several offers to relocate to the Bay Area all I can really say is that almost nobody pays well enough to make it worthwhile. You can make $150k-200k in Seattle, Portland, Denver, Austin, etc. and work on some really interesting problems with marketplace leaders (often even for companies based in the Bay Area). You end up living like a king comparatively. You'd have to make at least 30-40% more than that in the Bay Area just to maintain a lateral quality of life.

Having now screened over 300 candidates for openings on my team, the overwhelming majority of them being people already in the Bay Area, about the only thing I can say is that with all the easy money sloshing around the economy there it's just way, waaaaaaay to easy to be a terrible engineer and still maintain being a highly paid "professional". Most of the people I screen/interview are advanced-IDE users at best or simply framework operators (i.e. they have no idea how the things even work that they're claiming to be experts in).

Sure there are a lot of bright folks too (law of averages and all that), but it's depressing to see what people are getting paid to do glorified web scripting or whose entire job function could be replaced with 50-lines of Bash.

I'm extremely drawn to funny women. I like that they play around mentally with cultural taboos and find creative ways of walking the line between making themselves laugh and making other people uncomfortable.

You can now do this in a sanctioned way in Erlang as of v18. They're including merl as part of the standard distribution now.

Seems reasonable to me... except I don't see the point of having foo/1 and foo1/1. The author could just have easily done:

  foo(bar) -> void;
  foo(X) -> undefined.

  foo(X) -> foo1(X).

Your first paragraph is an expectations management problem on both sides.

That doesn't depend on the "technical" part of "technical leadership".

So "Elite Firms" basically have horrible hiring processes?

When I internalized this trend ~3 years ago, it's what spurred my interest in distributed storage systems across machines, racks, and datacenters, that are only memory backed.

The first versions of Objective-C were most definitely implemented as pre-processors to C compilation.

I'm a functional programmer (Lisp, OCaml, Erlang), and I only touch C++ when I have to wrap it in something to interface with a higher-level language.

Most of the cases where I run into this problem and end up feeling like I didn't gain much, if anything, from doing a straight C implementation are situations with deeply nested data structures.

I get that the thinking is that the benefit is that once I've made the compiler stop complaining, my memory management model should at least be sound and safe, and that is a win. Though immediately following that I start to have dream-like fantasies where this entire static analysis stage is simply bolted onto C instead of being a whole new language.

Afterall you can do some pretty impressive stuff with nothing but a pile of preprocessor macros (see Objective C).

This has been my problems with Rust for the last 10-12 months.

Lifetime elision seems thoroughly broken or extremely limited in functionality, and now when I'm writing (sometimes seemingly trivial) solutions in Rust I'm spending at least as much time and mental energy explicitly annotating lifetimes as I would be if I was just managing malloc and free in C.

As a result the pain vs. benefit curve doesn't bend nearly as far toward Rust as it theoretically should.

I realize they called this out in the article as "snark", but it really is just a blog engine, and its monthly web traffic is a small fraction of what wordpress.com gets (which is being served on one of the most craptacular platforms that exists).

They in fact could have solved this problem with Rails or even a more full-fledged CMS like BrowserCMS.

That's a terrible Faustian Bargain. The choice between someone who doesn't want to learn new things and someone for whom everything is new.

This is actually a good idea I think. Salesforce Chatter is a thing, and it sells well, and it's also terrible.

An on-prem corporate twitter would actually be useful.

But it's a SPoF and an extra remote or local service that has to be configured, maintained, etc. The thing that's coming online has to publish a message to redis anyway, so why not just maintain the service map inside the Rust application and open up a socket that allows those things to speak straight to Nucleon for mutating its configuration?

This has a redis service dependency? That is weird.

Fast start times are a valuable thing for a service component.

Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens.

Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.

TCP would guarantee delivery, but you're right in that you wouldn't know if the consumer actually did anything with the message. It could have crashed on parsing or something.

But moving the concern to the consumer to track the cursor doesn't make the protocol any more stable. To keep a stable cursor, the consumer would need to persist that someplace, which just pushes the acknowledgement to that persistence component instead. If a stable cursor is what you're after, then co-locating it with the durable queue provides a simpler solution with a slightly better consistency guarantee.

The garbage collection problem is a real one, but realistically how many consumers is an infrastructure service like this going to have? Tens? Hundreds? Thousands? Millions? Billions?

No matter which one of those you pick it's a trivially small secondary index to maintain even if you never reaped it. I mean it's a K/V problem (consumer_id -> queue_offest) and there's a K/V store already sitting there. If you didn't want it to grow forever then you could establish a TTL policy via configuration.

The problem you would have is consumers that don't have stable or bounded id's. Like a system that assigns a new id every time the consumer makes a request or the consumer is restarted.

Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs.

But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.

Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.

If you wanted the consumers to be stateless, assuming they otherwise had a deterministic identity, then you could have the queue operate like a journal internally, but present a unified queue API to consumers.

So the queue keeps track of the high-watermark on a per consumer basis and all the consumer has to do is show up, tell the queue its deterministic name/id (might be driven by imaging, configuration, or SDN), and the queue will serve up the next new item that consumer hasn't seen yet.

This would be handy for really dynamic transient worker topologies because it keeps the mutable state and state tracking concerns entirely outside the transient worker.

That said, I still wouldn't use LevelDB. Unless I was expecting to do multi-attribute range queries or something (now we're well outside queue territory), but even then you're still folding over the data for knowable start/end markers and a linear scan over a binary term file will be faster than the multiple seeks + segment scans that LevelDB requires.

LWPs and OTP patterns are a "parallelization framework". You have to try pretty hard to design an OTP application that can't be trivially parallelized, and that includes parallelization across machines, not just cores.

So you reinvented the vnode from riak_core?

This is one of the things I love about the Erlang ecosystem (Erlang, Elixir, Lisp-flavored Erlang).

Since the whole thing was designed to be an OS unto itself, it has the concept of "Applications" and "Releases" that come from OTP.

It's trivially easy to make things properly decoupled by dividing logical functionality into "Applications" and making a game-time call about whether or not they should be bundled in the release and consumed sorta like a library or run on a completely different VM/node and consumed like a service. You're architecturally incentivized to make "microservices", but fully able to couple or decouple them at will while maintaining the same semantics in how you consume them.

I don't really understand what the heavy emphasis on "real-time" here.

I mean it's log/event aggregation for ops insight. Unless the whole system is some tightly coupled feedback loop into an unsupervised machine learning model where the whole thing has actual hard-real-time requirements (something which might well be impossible to build), then there's no possible way that having a second or two delay, between a message being created and when you can actually see it, can possibly matter.

I mean you don't have someone with instantaneous reflexes and resolution ability sitting there 24/7 with their eyes peeled as a stream of thousands of log messages flies by.

The whole premise seems spurious. Is it really necessary for every startup and their uncle to delude themselves into thinking that their use case is "mission critical" or "carrier grade" or whatever?

Auth0 provides basically "login as a service". Its not like they're managing the access control to nuclear launch codes or something.

Unless some medical device manufacturer was stupid enough to make a critical surgical assistance device require an internet connection, a WAN round-trip on unreliable networks, and reliance on a 3rd party service in order to start operating it... how can this service being down possibly be anything more than an annoyance? What's the worst possible scenario? A session has to be rebuilt? A user has to make an extra login attempt?

By their own admission the service has gone down already due to the old system architecture. How many babies died?

I take it out for dinner and drinks. Spend some time getting know about it and where it comes from and what it does for a living. Then after we're a few cocktails in we get all philosophical. Really start asking the hard questions like, "Why do I even exist? Is any of this real or is it all some weird virtual world?"

We become fast friends and feel like we really understand each other.

But days pass, and each encounter feels less magical. It's almost like we having nothing in common. Like we're from two completely different worlds. One where its stuck in the past and one where I'm ambitious and excited about the future.

After awhile we don't really speak to each other anymore, and after some pretty ugly fights at work that get too personal... I rewrite it.

lol, this reads like the way the world ended up with MongoDB.

There are some things you have to do right from the beginning or you're never going to get them right.

I'd say my problem with the whole massive containerization hype circus is less that I'm a curmudgeon who hates anything new, and more of a practicioner who hates marketing and social hype promoting a half solution to a narrow problem as a full solution to all problems.

Containerization isn't new. It just has a brand name now, and the "all the way" solution to this problem is unikernels.

Because I already got burned once paying for TextMate and having the development basically die. The same pattern seems to be happening with Sublime Text.

So I finally just started using Emacs instead. Setup my keybindings so they're Mac/Sublime Text like and wrote a little custom frame/window manager in elisp to make the application behave the way I want, and I'm free from having to worry about one of my core tools turning into abandonware.