dmw_ng

Fixed just for you :) I always forget that <meta> tag.

I'm a restic user, but have resisted the urge to attempt a bikeshed for a long time, mostly due to perf. It's index format seems to be slow and terrible and the chunking algorithm it uses (rabin fingerprints) is very slow compared to more recent alternatives (like FastCDC). Drives me nuts to watch it chugging along backing up or listing snapshots at nowhere close to the IO rate of the system while still making the fans run. Despite that it still seems to be the best free software option around

That's been a feature of S3 for quite a long time now, called S3 Select https://docs.aws.amazon.com/AmazonS3/latest/userguide/select...

Despite it being an awesome feature I've been itching to use, I've never actually found a use for it beyond messing around. Most places where S3 Select might make sense seems to be subsumed (for my uses) by Athena. Athena has a rather large amount of conceptual and actual boilerplate to get up and running with, though, S3 Select requires no upfront planning beyond building a fancy query string (or using their SDK wrappers)

Where S3 Select is likely to become fiddly is anywhere multiple files are involved. Athena makes querying large collections of CSVs (etc) straightforward, and handles all the scheduling and results merging for you.

Modern .net on Linux is lovely, you can initialize a project, pull in the S3 client and write a 1-3 line C# program that AOT compiles to a single binary with none of the perf issues or GIL hand-wringing that plagues life in Python.

Given modern Python means type annotations everywhere, the convenience edge between it and modern C# (which dispenses with much of the javaesque boilerplate) is surprisingly thin, and the capabilities of the .net runtime far superior in many ways, making it quite an appealing alternative especially for perf sensitive stuff.

I recently bought an n100 and within a matter of days got buyer's remorse and impulse-purchased an n305 to go right beside it, which is currently sitting with a wildly overpriced 48 GB stick installed and 2TB SN850X, it's an absolute joy perfwise and the absence of heat it generates.

The only thing I'd reserve judgement on is the tendency to throttle. I haven't got far enough to characterize it, but it's not clear how much value those extra cores will add over the n100 with TDP settings tweaked down in the BIOS, and if leaving the n305 to run at max TDP, heat/noise/cost/temperature-related instability may start to become an issue, especially when packing other hot components like a decent SSD into the tiny cases they come in.

Seems like a massive distraction from their offering for a small company, wonder why they didn't consider something like tight integration with OnlyOffice or similar. Setting out to build a new office suite feels about as sensible as building a new web browser from scratch. Except at least with a browser, you have open specs helping you through most of the endless supply of compatibility problems.

converting huge amount of xml files

pickling

Sounds like if this is the tooling and the task at hand, about the most complex things that should be passing through the pickler are partitioned lists of filenames rather than raw data. E.g. you can have each partition generate a parquet for combining in a final step (pyarrow.concat_tables() looks useful), or if it were some other format you were working with, potentially sending flat arrays back to the parent process as giant bytestrings or similar

This is not to say the limitations don't suck, just that very often there are simple approaches to avoid most of the pain

It's comical to see the sudo codebase mentioned in the same breath as increasing auditability here

Sufficiently fast software often allows leaving out whole layers of crap and needless indirection, the most common being caching. Fixing an algorithm so you can omit a dedicated database of intermediate results can be a huge maintainability/usability improvement. The same principle appears all over the place, e.g. immediate mode UIs, better networking (e.g. CSS image tiling vs. just fixing small request overhead in HTTP1 vs. QUIC), importing giant CSV files via some elaborate ETL process vs. just having a lightning fast parser+query engine, etc.

Depending on how you look at it, you could view large chunks of DOM state through this lens, as intermediate data that only exists to cache HTML parsing results. What's the point of allocating a hash table to represent element attributes if they are unchanged from the source document, and reparsing them from the source is just as fast as keeping around the parsed form? etc. These kinds of tricks only tend to present themselves after optimization work is done, which is annoying, because it's usually so damn hard to justify optimization work in the first place.

I've had similar times with DuckDB, it feels nicer to use on the surface but in terms of perf and actual function I've had a better experience with clickhouse-local.

SPF+DKIM+DMARC are a classic case of Goodhart's law, the amount of spam they stop these days (at least anecdotally) is minimal. Most spam I get seems to come via SalesForce infrastructure, and a variety of similar bulk email marketing providers

Linux with HiDPI, is such a chore

There's a simple broad-spectrum fix for that. For the odd occasion I want to view high definition photos in Linux I just switch screen mode temporarily, for the rest it is sooo not worth the hassle

CloudFlare does not provide unmetered anything, at best they provide services on a discretionary basis while trying as hard as possible to make it appear this is not the case. It's better to think of their product line as a CRM system with some CDN features on the side

do lots of useful work for free

Have often wondered about this in terms of some of their control plane APIs, a read-only IAM key used as part of C&C infrastructure for a botnet might be interesting, you get DNS/ClientHello signature to a legitimate and reputable service for free, while stuffing "DDoS this blog" e.g. in some tags of a free resource. Even better if the AWS account belonged to someone else.

But certainly, ability to serve an unlimited URL space from an account with only positive hits being billed seems ripe for abuse. Would guess there's already some ticket for a "top 404ers" internal report or similar

I prefer your version: Barr replies to a tweet before gatecrashing the next S3 planning session. "A customer is hurting, folks!". The call immediately falls silent with only occasional gasps heard from stunned engineers, and the gentle weeping of a PM. I wonder if Amazon offers free therapy following an incident like this

Can't imagine a change like this would be made without some analysis.. would love an internal view into a decision like this, I wonder if they already have log data to compute financial loss from the change, or if they have sampling instrumentation fancy enough to write/deploy custom reports like this quickly.

In any case 2 weeks seems like an impressive turnaround for such a large service, unless they'd been internally preparing to acknowledge the problem for longer

He did not "win in court", a consent order means both parties agreed to a mutual resolution prior to a judgement occurring. If I had to guess it was settled this way because of the time, misery and tremendous cost that would otherwise be involved for all parties. "Wins in court" suggests some thorough process leading up to a final evidential determination made by a judge. That is not what happened here, it'd probably be more accurate to say the parties were motivated to cooperate via their solicitors under looming threat of enduring that process.

No skin in this game, but looking at it from the respondents' perspective, 20k split between 4 is incredibly attractive: 5k and an apology to put an end to any threat relating to the issue, given the alternative of potentially unbounded costs to demonstrate innocence with a tortuous and uncertain outcome. From the claimant's perspective it is more confusing, triggering such a heavyweight process then settling for so little given the claimed harms suggests all kinds of things, not least including the kind of advice he may have received given the strength of the case as his solicitors understood it.

We only very briefly had that around the start of Win95, by the time 98 came around (IIRC) Explorer and definitely IE already had flat toolbar buttons that only had bevels when hovering. I remember fetishizing how clean those new style controls looked (esp IE5ish with its grey-on-grey patterned background texture!) but it definitely broke the consistent interaction idiom you mention

I'm not promoting one way or the other, just pointing out why things are the way they are. Restarting a service with stateful networking is reason enough to avoid it where possible, watching entire buildings melt for 15 minutes because a single binary SEGV'd is a real outcome. For an extra helping of pain, add a herd of third party clients of random versions to a system that never needed the comms capabilities on offer and you have a problem to solve that never needed to exist in the first place

There's a reason pretty much everything that does not require low latency replies avoids stateful networking - everything from RSS to video streaming prefers stateless polling designs because it is vastly easier for both parties to implement and scale. Meanwhile, I couldn't name a single system in widespread use built around a MQ paradigm in its public interface, except for actual MQ APIs, and many of those (e.g. from AWS) are still built on polling for the reasons just described

AWS already give you intelligent tiering for this, it's a very nice product but it's also just a nice way of hiding the same fees. Your $0.004/GB becomes $0.023/GB on first read for 1 month then $0.0125/GB for 2 months, so the average cost of storing it over those 3 months becomes $0.016/GB, and that's before considering monitoring fees

At least magnetic disks are iops constrained, lower iops loads conceivably allow higher density, or packing different load patterns to the same devices. Say a 8 TB / 100 iops disk reserves 90 iops for a 1 TB a database service, that's 87% of the disk's capacity sitting free but only 10 iops to serve it with. Adding what is effectively an iops tax to discourage frequent reads is one way to make a mixture like this work (or another way to think of it - subtracting an iops discount)

Obviously example above is contrived, but same principle applies to a pool of 1000 disks as it would 1. You also don't escape this issue with regular hot storage either, there is still a (((iops * replication count) / average traffic) / max latency) type problem lurking, which would still necessitate either limiting density or increasing redundancy according to expected IO rate. This is one reason why some S3 alternatives with weaker latency bounds (not naming names, they're great but it's just not the same service) can often be made substantially cheaper, and why at least one of S3's storage classes may be implemented entirely as an accounting trick with no data movement or hardware changes at all

Try saying it out loud with 1 TB to recompress and host for a fraction of users to save some CPU, because if we're talking CPU we're talking latency, and there's not much point in changing if you're running zstd --ultra --long -22 for every request

A very happy moment for Yann no doubt, he mentioned Google's competing compression efforts directly impeding his work during a corecursive podcast interview ( https://corecursive.com/data-compression-yann-collet/ )

For the rest of us, another half-decade+ wait to get useful levels of compatibility especially on mobile

My best guess is their primary product is a Windows (I think) backup client that has a place for smarts allowing them to paper over problems, or something along those lines. Feels like an "oh Germany is down again, when is Frank back from holiday so he can catch a plane to replace the switch?" type affair. If you google around the Reddit data hoarder communities you'll find plenty of war stories about E2

Nice to see this idea written about in detail. I had thought about it in the context of terrible availability bargain bucket storage (iDrive E2), where the cost of (3,2) erasure coding an object and distributing each segment to one of 3 regions would still be dramatically lower than paying for more expensive and more reliable storage.

Say 1 chunk lives in Germany, Ireland and the US each. Client races GETs to all 3 regions and cancels the request to the slowest to respond (which may also be down). Final client latency is equivalent to that of the 2nd slowest region, with substantially better availability due to the ability to tolerate any single region being down

Still wouldn't recommend using E2 for anything important, but ^ was one potential approach to dealing with its terribleness. It still doesn't address the reality of when E2 regions go down, it is often for days and reportedly sometimes weeks at a time. So reliable writing in this scenario would necessitate some kind of queue with capacity for weeks of storage

There are variants of this scheme where you could potentially balance the horrible reliability storage with some expensive reliable storage as part of the same system, but I never got that far in thinking about how it would work

it is rumoured Flash Crabs will be out in the fall

I didn't read heroics in the above tweet and I'm more than a little confused about the sour grapes here. Is it too much to forgive inaccuracies in a 29 year old memory for such a low stakes issue, especially when there is a huge benefit to giving a personality to some ancient OS history?

It wouldn't matter if he claimed his 5.25" drive shot lightning bolts the first time he tried to compile that dialog, the stories are still great and it's personally enough for me having grown up with these systems to hear them told (not much different from listening to a pseudo-senile grandparent exaggerate war stories for entertainment's sake)

Tech has all the attention span of a goldfish https://techcrunch.com/2019/02/21/redis-labs-changes-its-ope...

There are tons of resource types only available via cfn compared to cloud control api vs terraform vs direct api, any decently complex deployment often requires mixing one or more options just to get full coverage