dormando

Does anyone run these "at home" with small clusters? I've been googling unsuccessfully and this thread doesn't refer to anything.

So a non-quantized scout won't fit in a machine with 128GB of RAM (like framework or mac studio M4). Maverick is maybe a 512GB M3 Max mac studio. Is it possible (and if so what're the tradeoffs for) running like one instance of Scout on three 128GB frameworks?

Half on topic: what libs/etc did you use for the animations? Not immediately obvious from the source page.

(it's a topic I'm deeply familiar with so I don't have a comment on the content, it looks great on a skim!) - but I've been sketching animations for my own blog and not liked the last few libs I tried.

Thanks!

Throwing out a clarification: EVcache is effectively a complex memcached client + an internal ecosystem at Netflix. You can get much of its benefits with other systems (such as the memcached internal proxy: https://docs.memcached.org/features/proxy/).

For plugging into other apps they may only need a small slice of EVCache; just the fetch from local-then-far, copy sets to multiple zones, etc. A greenfield client with the same backing store could be trivial to do.

That all said I wouldn't advise people copy their method of expanding cache clusters: it's possible to add or remove one instance at a time without rebuilding and re-warming the whole thing.

:'(

At some point the OBSD CVS server was something I donated. Don't think it's in this picture as I don't think it was a Dell. Never saw a picture of the thing in action actually, doh.

I avoided spending a quarter mill on F5's by deploying OpenBSD as firewalls/L4 routers but I was able to keep some of the budget... For a year there I would e-mail them every 6mo and ask what they wanted. Sad when I changed jobs and had to stop.

I just got one a few weeks back but haven't gotten to spend a ton of time with it yet. It's taking some adjustment but I'm liking it so far.

- had 1440p + 1080p monitors on stands side by side before. Now just this one on an arm (which is excellent), that I can adjust to keep my position from being static.

- not having to hold my neck angled while reading my side monitor is helpful.

- realistically there are a few "modes" of working on here. While coding it's pulled a bit closer, while in CAD or similar creative I might push it back a bit and get more of the monitor in view.

- I recline slightly so the monitor is tilted a bit which gives me a solid view of the bottom 60-70% of the monitor. The top is a bit out of range at close distance.

- For coding so far I have the middle-ish of the monitor as a 1440p code-only view. Below that are a few windows for manpages/reference/interactive debugging/repl/etc. On the top end which is normally slightly out of view I have compilation and long running test output which I glance at by moving my eyes.

I like not having to page between desktops while coding when possible. The bottom view is also large enough to hold a browser window or simulator window. Need to also try pushing it back a bit with slightly larger text and see if that's any better.

I don't intend to game on it, maybe windowed mode in the middle or something.

edit: well, also it has this mode where you can split it into two 1440p monitors on different inputs (which you can hook up to the same computer), so depending on the game I might do that as well.

client ecosystem is definitely a sore point now. I've just sort of started working on a replacement for libmemcached to hopefully cut down on the complexity... but then that's a migration and nobody wants to do that.

Pinterest should drop me a line if they're interested in sponsoring work though :)

I can say with certainty this isn't strictly true. The failures should be relatively rare; when I say relatively I mean on the level of natural node failure. If natural node failure isn't survivable without special systems to quickly replace downed nodes you don't actually have an N+1 redundancy system. Thus, the pools aren't large enough :) Or, in this case, if they really are failing this much then having them always lose their cache is a major reliability hole.

It's a subtle difference. I think many operators get used to node failures being extremely common when they don't necessarily have to be. I suspect the note on "if they come back on their own ensure they're flushed" meaning they have something unusual causing ephemeral failures. If that's just "cloud networking" there isn't much they can do but it's almost always fixable.

Now a more philosoraptor style comment: I see Mcrib is a service built to quickly detect and replace memcached's. I treat memcached in infrastructure as a very stable service. Meaning it is infrequently necessary to upgrade it, and it will generally not fail on its own. If it does it will be highly infrequent compared to services with higher churn or more complexity/dependencies. This means if they're failing often enough that you need to rapidly detect and replace them you have a more fundamental problem.

From a structural standpoint I think my technical comment can be useful. If things really are failing this much A) you should figure out why and slow that down. B) if you have a generally stable system and understand the typical rate of failure, you can add tripwires into Mcrib to avoid over-culling services and loudly raise alarms. Then C) you can improve technical reliability with redundancy/extstore/etc.

I've also seen plenty of times where folks have a dependency of a service determine if that service is usable, which I disagree with quite strongly. Consul being down on a node should trigger something to consider if the service is dead. It's important both for reliability (don't kill perfectly working things because you end up having to design around it), and for maintainability as you've now made people afraid of upgrading Consul or other co-dependent services. Other similar failures are single-point-of-testing availability checking where instead you probably want two points of truth before shooting a service.

Now you risk people being afraid of upgrading probably anything, which means they will work around it, abstract it, or needlessly replace it with something they feel safer managing. The latter is at best a waste of time, at worst a time bomb until you find out what conditions this new thing breaks under.

This isn't advocating that you design without assuming anything can fail anywhere at any time; just pointing out that how often a service _should_ fail is extremely useful information when designing systems and designing fail safes, alerts, monitoring, etc.

Hi! I'd like to offer some hopefully useful information if any Slack folks end up reading this, or anyone else with a similar infrastructure. I'll start with some tech and make a separate philosophical comment.

Also caveat: I have no deep view into Slack's infrastructure so anything I say here may not even be relevant. YMMV.

First some self promotion: https://github.com/memcached/memcached/wiki/Proxy memcached itself is shipping router/proxy software. Mcrouter is difficult to manage and unsupported. This proxy is community developed, more flexible, likely faster, and will support more native features of memcached. We're currently in a stabilization round ensuring it won't eat pets but all of the basic features have been in for a while. Documentation and example libraries are still needed but community feedback help speed those up tremendously (or any kind of question/help request).

It's not clear to me why memcached is being managed like this; mcrouter seems to only be used to abstract the configuration from the clients. It has a lot of features for redundant pools and so on. Especially with what sounds like globally immutable data and the threat of cascading failures during rolling upgrades it sounds like it would be very helpful here.

If cost or pool sizes are the main reasons why the structure is flat, using Extstore (https://github.com/memcached/memcached/wiki/Extstore) can likely help. Even if object value sizes are in the realm of 500 bytes, using flash storage can still greatly reduce the amount of RAM necessary or reduce the pool size (granted the network can still keep up) with nearly identical performance. Extstore takes a lot of tradeoffs (ie; keeping keys in RAM) to ensure most operations don't actually write to flash or double-read. Extstore's in use in tons of places and everyone's immediately addicted.

Finally, the Meta Protocol (https://github.com/memcached/memcached/wiki/MetaCommands) can help with stampeding herds to help keep DB load from exploding without adding excess network roundtrips under normal conditions. I've seen lots of workarounds people build but this protocol extension gives a lot of flexibility you can use to help survive degraded states: anti-stampeding herd, serve-stale, better counter semantics, and so on.

License?

The venerable lodev tutorial uses this method, which I also used for most of my engines. I learned an interesting tidbit while comparing the two methods though:

The old-school original methods used pretty small cos/sin/atan lookup tables to do the ray and then the correction calc. Using the linear method you end up with a couple divisions per ray that aren't there in the lookup method. Divisions were (and are, depending on the platform) pretty slow. Linear method still works with lookup tables but they're relatively huge.

Also IIRC With the linear method door-indents need a workaround.

Looks like the engine is missing the fish-eye correction in the ray-cast calc. I love writing these engines for fun :)

It was an algorithmic/lock scaling limit. Originally it was single threaded, then when it was first multi-threaded it scaled up to 4 threads. Then I split up some locks and it scaled to 8 threads (depending). Then I rewrote the LRU and now reads mostly scale linearly and writes don't. If there's enough interest we'll make writes scale better.

Partly this is because the software is so old that the thread scalability tends to track how many CPU's people actually have.

Everything that interacts with the disk is extstore.c, most of the wrapper code that glues memcached with extstore is storage.c. extstore.c has barely changed since I first wrote it; so there's not much maintenance overhead vs mmap anyway.

That's an excellent question; it turns out there are a _lot_ of semantics that the OS is covering up for you when using mmap. For instance (this may be fixed by now), but any process doing certain mmap syscalls locked access to any open mmap's in an OS. So some random cronjob firing could clock your mmap'ed app pretty solidly.

There are also wild bugs; if you google my threads on the LKML you'll find me trying to hunt down a few in the past.

Mainly what I'm doing with extstore is maintaining a clear line between what I want the OS doing and what I want the app doing: a hard rule that the memcached worker threads _cannot_ be blocked for any reason. When they submit work to extstore, they submit to background threads then return to dequeueing network traffic. If the flash disk hiccups for any reason it means some queue's can bloat but other ops may still succeed.

Further, by controlling when we defrag or drop pages we can be more careful with where writes to flash happen.

TLDR: for predictable performance. Extstore is also a lot simpler than it may sound; it's a handful of short functions built on a lot of design decisions instead of a lot of code building up an algorithm.

There are stats in "stats items" / "stats slabs". Last access time for most recent eviction per slab class, etc. (see doc/protocol.txt from the tarball).

"watch evictions" command will also show a stream of details for items being evicted.

Looks like this article got one bit of updated information but missed everything else... I'll address some things point by point:

data structures: - yes, fewer datastructures, if any. The point isn't that they have the features or not, but that memcached is a distributed system _first_, so any feature has to make sense in that context.

"Redis is better supported, updated more often. (or maybe memcached is "finished" or has a narrower scope?)" - I've been cutting monthly releases for like 5 years now (mind the pandemic gap). Sigh.

Memory organization: This is mostly accurate but missing some major points. The sizes of the slab classes doesn't change, but slabs pages can and do get re-assigned automatically. If you assign all memory to the 1MB page class, then empty that class, memory will go back to a global pool to get re-assigned. There are edge cases but it isn't static and hasn't been for ten years.

Item size limit: The max slab size has actually been 512k internally for a long time now, despite the item limit being 1mb. Why? Because "large" items are stitched together from smaller slab chunks. Setting a 2mb or 10mb limit is fine in most use cases, but again there are edge cases, especially for very small memory limits. Usually large items aren't combined with small memory limits.

You can also _reduce the slab class overhead_ (which doesn't typically exceed 5-10%), by lowering the "slab_chunk_max" option, which puts the slab classes closer together at the expense of stitching items larger than this class. IE; if all of your objects are 16kb or less, you can freely set this limit to 16kb and reduce your slab class overhead. I'd love to make this automatic or at least reduce the defaults.

LRU: looks like the author did notice the blog post (https://memcached.org/blog/modern-lru/) - I'll add that the LRU bumping (mutex contention) is completely removed from the _access path_. This is why it scales to 48 threads. The LRU crawler is not necessary to expire items, there is also a specific thread that does the LRU balancing.

The LRU crawler is used to proactively expire items. It is highly efficient since it independently scans slab classes; the more memory an object uses the fewer neighbors it has, and it schedules when to run on each slab class, so it can "Focus" on areas with higest return.

Most of the thread scalability is pretty old; not just since 2020.

Also worth noting memcached has an efficient flash backed storage system: https://memcached.org/blog/nvm-caching/ - requires RAM to keep track of keys, but can put value data on disk. With this tradeoff we can use flash devices without burning them out, as non-get/non-set operations do not touch the SSD (ie; delete removes from memory, but doesn't cause a write). Many very huge installations of this exist.

I've also been working on an internal proxy which is nearing production-readiness for an early featureset: https://github.com/memcached/memcached/issues/827 - scriptable in lua, will have lots of useful features.

It's not really realistic, you're right. For my own goals it's "defense in depth" - just because I can't think of a scenario now doesn't mean it's impossible to do. Access also makes it easier to accidentally configure it in a way that is in fact easy to blow up.

From a practical standpoint, I just don't want any not-me traffic hitting the management interface for any reason (intentional or not), as I assume they're poorly written and can easily be crashed or even bricked. I've locked myself out of very expensive enterprise switches in past lives by ssh'ing to them too many times.

So if IE someone can poke my management VLAN by sending an ICMP packet with a spoofed return address and my RPI doesn't filter that right because I did something wrong... I'm happier if that can't tickle the management interface at all.

Hope that works :) I have this set up to an AT&T fiber gateway trashcan in pass-thru mode, so technically the RPI's vlan port has a public IP address. Otherwise I couldn't get upnp/etc to work when I wanted to.

I also want to be able to set up a DMZ'ed VLAN to hook up an old NUC to host something like a valheim/minecraft/whatever server if I wanted. So having the VLAN be safe was a goal for me.

fwiw when I bought it, it was on sale for ~$50 too. couldn't find a 2.5g switch that I liked in my price range, so went cheap as possible and will swap it when a 2.5g hits all my requirements :)

Did this earlier in the year with an rpi4 and a netgear managed switch. It can route 780mbit down/up (nftables)! Can probably get closer to 900mbit with some overclocking, and reduce latency slightly by pinning the rpi's clock but haven't really cared to.

The hard part was finding a cheap managed switch which does _not_ expose the management interface on every vlan. This is a specific product feature (wtf?) and often the only difference between two product lines.

Ended up with a netgear GS308T - the internet is full of rage at the device requiring you to log into netgear and register it, but I can confirm that I didn't have to do anything. I set it up with it connected directly to a laptop and have never signed up for a netgear account, though they may lock this down in the future. I was even able to upgrade the firmware. Would've preferred a model with ssh access but tp-link wasn't selling them at the time?

Wanted to be able to mix multiple ISP's together (but ended up only having one for now), which would've been 2+ USB NICs and not great for perf.

The other reason why I ended up here is because OpenWRT/etc support for wifi6 routers wasn't/isn't coming, and I want to be able to place my WAP in a different location than the router and main switch... so the WAP is a wifi6 thing running stock firmware but in "WAP mode" which turns everything off.

Anyway, works fine. Image your SD card when you're done configuring it. I need to go back and put it in read-only mode but haven't cared.

This comment is correct. I made that change ages ago. Amused it's still there.

https://twitter.com/dormando/status/1402466173778677764

my fault. I would sometimes monitor for "guru mediations" popping up to tell if we were throwing errors without it being caught by other systems. Among other reasons.

Reads-wise I've done tests with up to 48 cores... It scales just about linearly. Writes don't scale much past a single core but really nobody asks for this so I haven't worked on it.

For fun I did a test benchmarking misses, getting 60 million rps on a single machine :) For my high throughput tests the network overhead is so high, that to discover the limit of the server the benchmark client has to be run over localhost. Not terribly useful; most people's networks will peg well before the server software. Especially true if your objects aren't tiny or if you batch requests at all.

I've yet to see anyone who really needs anything higher than a million RPS. The extra idle threads and general scalability help keep the latency really low, so they're still useful even if you aren't maxing out rps.

You can see tests here too: https://memcached.org/blog/persistent-memory/ - these folks might dismiss this testing as "not a cache trace", but I don't feel that's very productive.

Specifically to the cache traces though, that's just not how I test. I never get traces from users but still have to design software that /will/ typically work. Instead I test each subsystem to failure and ensure a non pathological dropoff. IE; if you write so fast the LRU would slow you down, the algorithm degrades the quality of the LRU instead of losing performance; which is fine since in most of these cases it's a bulk load, peak traffic period, load spike, etc.

I've seen plenty of systems overfit to production testing where shifts in traffic (new app deploy, new use case, etc) will cause the system to grind to a halt. I try to not ship software like that.

All said I will probably try the trace at some point. It looks like they did perfectly good work. I would mostly be hesitant to say it's a generic improvement. I also need to do up a full blog post on the way I test memcached. Too many people are born into BigCo culture and have never had to test software without just throwing it in production or traffic shadow or trace. I'm a little tired of being hand-waived off when they run in one use case and mine runs on many thousands.

At risk of making a useless comment: "nice".

Local memcached's? local caches or still distributed cache?

Ah yes, I misremembered: there's an additional slot but a very specific spec requirement. They'll be whitelisting individual devices.

They have mentioned a few times (though nothing with specific numbers) that games have a lot of data redundancy to work around poor HDD seek times.

Given the drive speed it might be nice if games could auto-archive into a USB-attached device. The XBox requires custom SSD hardware for expansion (those weird non-USB drives?) and the PS5 will require swapping out the SSD entirely to expand usable space.

... but sequentially copying a compressed archive to a fast USB3.1 device might not be so bad for games not currently in use.

These are fun!

Shameless: http://www.dormando.me/post/fpga-raycaster/ - I did one of these last year with an arduino using an FPGA as a weird GPU :)

Fwiw this post comparison is pretty old. The memory usage should be the same or better these days (automatic slab rebalancing, LRU expired item crawler, chunked items, etc).