noirscape

I want a stable platform that will host my small static website blog at a very low cost, with high stability and uptime, no questions asked.

Maybe try the neocities supporter plan[0]? They allow custom domains, offer decent bandwidth and put a lot of effort into open source and encouraging people to write their own sites from what I can tell.

It seems like a better option than dealing with the Great Amorphous Corporate Centralizing Mass that is Cloudflare.

[0] https://neocities.org/supporter

It's because of Google Play Services and the subsequent years long gutting of AOSPs application list. Google enforces OEMs to bundle their entire suite if they want to ship with Play Services and then uses that as an excuse to kill the basic AOSP phone apps that are outside of their ecosystem. That in turn harms the ability for Custom ROMs to ship basic apps like SMS readers, phone dialers and similar such, all of which used to be (maybe some still are, but they're all very much not well maintained) part of AOSP. If there's any silver lining, the upcoming play service blockade to install your own software on Android is likely going to at least create a burst of users for older devices.

The actual numbers are frankly even more disappointing - these numbers are heavily pushed up up by waydroid, which is an emulator for Android apps on Desktop Computing. More than half of the US installs is running Waydroid, with the actual most used real device in the US being nx_tab... which is LineageOS for the Nintendo Switch. That's a difference of 180k and 12k btw. The most used actual phone in the US for LineageOS is beyondx, which is the codename for the Galaxy S10 5g, a device that at a glance stopped being sold last year.

China by contrast fares much better when it comes to LineageOS (as Google Play Services isn't allowed in the country by export controls, the control from Google isn't nearly as strong there); the most used device there is actually a phone, the Xiaomi Mi 8 (dipper) and right behind it, the Xiaomi Mi 10T(/Pro) codenamed apollon.

Final country worth mentioning is Brazil, which apparently really likes the moto g7 power (ocean), a phone from roughly the same period as the Galaxy S10 5g.

Vietnam is also relatively high, with the Galaxy S7 (herolte) being the most used device. Russia is just a case where it's basically all waydroid users - not real phones.

At least from my understanding of the world, most of these numbers make sense if you consider them in proximity to US power, financial capabilities making phones last longer than their official support dates and just a rough idea of what phone brands are popular in which country.

Because the conversation is incomplete if we're talking internationally.

The cost of living in the US is much higher compared to most other first world/rich countries for one. Count up someone's basic living expenses in the US and those in another country (so taxes, rent and fixed costs) and the US often ends up much higher in terms of absolute values. In other countries, taxes usually soak up more of those fixed costs, reducing them more across the board for most people. The US also has very little protection against surprise fees at checkout (to the annoyance of non-Americans when ordering stuff online from the US), so a lot of stores sell on higher markups relatively speaking, making the same goods more expensive in the US. There's also healthcare, which needs little elaboration because the US is to my knowledge the single most expensive country to live in when it comes to that.

That applies to the US as a whole; it's why someone can say they're making 300k USD a year, say they're apparently barely able to stay afloat and then the rest of the world pretty much regards the US economy as being fundamentally wrong in some form. In most places, 300k USD a year is living in the upper class (as in, "work this job for a decade and you can retire early" money), not scraping the bottom of the barrel. By modern conversion standards, that's about 263k euros, or about 21k euros each month.

Then there's the tech sector specific problems. San Francisco is expensive to live in, and most US tech companies are in SF. Take the US cost of living problem, amplify it specifically for the tech sector (which is usually not talked about, since it's hard to vocalize). Second is that the US tech sector has more creative ideas and money than business sense - throwing money at a problem like the purse doesn't exist is a very US tech thing that doesn't apply anywhere else. It means that it's possible to hire people at far more inflated prices than the job is realistically worth.

Whether a wage is good or bad is pretty much entirely dependent on the local economy. Someone making 2000 EUR a month in Europe makes just above/right below the poverty line. Someone making 2000 EUR a month in Brazil is living an upper class lifestyle. That's an extreme comparison, but is a good indicator.

You're probably thinking of the Steam debacle. Nintendo wasn't responsible for that.

What happened was that Dolphins developers wanted to release the emulator on Steam. Valve, independently from anyone else, send a message to Nintendo's legal team asking if they think it's permissible to distribute Dolphin on Steam. Nintendo's lawyers essentially responded with the company's policy on emulation ("third parties doing emulation is not okay") and that they might consider looking into their options should Dolphin release on Steam. After that, Valve told the Dolphin developers that the game was banned from Steam.

Nobody send any legal threats or anything; no C&D was issued, no DMCA invoked, no lawsuits, nothing. As far as the legal side of things is considered, the only thing that happened is that a business refused to do business with someone. (Which is generally their right to do, as long as it's not because that someone belongs to a protected class and being an emulation developer is not a protected class.) That's why Dolphin's devs also effectively had no recourse, even if they could pay the necessary lawyers. You can't force someone else to sell/publish your stuff.

Great post. Redis is just kinda overkill whenever I've had to use it. Memcached by contrast is very simple, fast and works without needing to do much fiddling with it.

One big tip I should recommend is to increase the default memory size limit to something more realistic for modern hardware (and arguably this should just be increased on the upstream's side as well, instead of making everyone reconfigure shitty defaults). It's very easy to exceed the memcached default key value, since it's just 1mb; the maximum size of memcached as a whole is 64mb, which is similarly very low. Outside of that, it works very well and the lack of persistence is great at making it not do things it's not supposed to do (which is a big problem with Redis' feature creep, the projects mainpage promoting AI drivel alone should point towards that.)

It's also worth noting that some of those trillion dollar companies have had staggeringly bad responses when confronted with the fact child predators are running amok on their platform.

The CEO of Roblox is probably the single easiest example to point at; when confronted about his platforms issues when it comes to enabling child abuse, the first response he had was to claim that child predators were an untapped market and then claim to be interested in adding a dating site feature to Roblox.

That's the kind of rethoric these bad laws are a response to, and is the elephant in the room that a lot of the tech industry fails to recognize. (Including the privacy advocates, for whom every nail looks like it has a hammer shaped solution.) Age verification isn't a good solution to this problem, but it at least forces the hands of these companies to address it if they don't want to face jailtime for knowingly abetting predators - they can't pretend to have clean hands anymore if they're mandated to verify user ages.

There's almost certainly better solutions, but that's also why attestation (where the source device transmits the user's age, rather than storing a ton of PII of them elsewhere) misses the mark. Attestation doesn't fix that problem.

The closest vision back then to what we're getting now is the moody ship's computer from hitchhiker's guide.

It's not the ship computer, but the door AIs, which had this marketing blurb in the brochure:

All the doors in this spaceship have a cheerful and sunny disposition. It is their pleasure to open for you, and their satisfaction to close again with the knowledge of a job well done.

Tellingly, the main characters respond with annoyance whenrver the doors speak up.

Hitchhikers Guide should not have been as prophetic as it ended up being, but here we are.

To be fair, the advice very rarely is for people to jump onto Arch based distros.

The problem is more that the Arch value proposition kinda presupposes the sort of user that's going to "feel superior" about having it installed[0]. It leads to people that have no business installing Arch Linux (as it doesn't match their usecase) installing Arch Linux because it makes them feel cool.

I don't have a good answer for this, besides making it more apparent what people should expect from having Arch installed. My recommendation usually goes something like this:

* Do you want to have the latest version of all software, regardless of the question if it's well-tested beforehand?

* Do you want to have all software distributed in an as-close-to-upstream approach as possible? Be aware that "upstream" configuration can sometimes significantly differ from defaults most people expect. (Sometimes there's reasons for this, sometimes upstream are a bunch of obstinate jerks.)

* Are you comfortable with a terminal?

* Are you comfortable with needing to suddenly learn how to troubleshoot a broken system after a routine update?

Only if the answer to all of those is "yes", then Arch is suitable for you.

And finally, more specific to servers, where the answer should be "no" if you want to use arch:

* Do you have the expectation to never have to touch the OS after it's been configured correctly besides routine maintenance (ie. installing security updates) and maybe a big update twice a year?

I used to use Arch, before realizing that my system was gradually morphing into a bespoke mess that didn't really serve my needs and that while doing something very specific was possible, I also had to configure a bunch of mundane stuff you aren't normally required to think about - there's never a "just install, activate and adjust as needed" with Arch. All I actually wanted was a distro with more recent software than "3 years old" (Debian/Ubuntu's sluggish package inclusion is not really useful for desktops).

So I looked around and realized Fedora worked better for me: professional, clean, recent software (every 9 months updates, feature freezes are smart enough to account for ie. New Python releases) and not prone to sudden surprises.

[0] https://wiki.archlinux.org/title/Arch_Linux is a good example of it.

Browsers have an absolute insane level of relatively unchecked permissions to do whatever they want on a client.

There's a lot of effort by browser developers to scope creep the browser into essentially being an OS-agnostic tech stack (one where, conveniently, code can be shipped across the network "as necessary", removing a lot of user agency for the software being ran); Chrome being the biggest driver of this, while Firefox has an extremely weak spine in trying to limit it.

It's fairly dire and I wouldn't be surprised if there's a lot more of these side channel attacks in a lot of web APIs.

They can, but there's an OS option that basically is "I'm going to say yes, but then effectively do no". Basically it'll pretend to the application that a permission is granted, but then just keep returning empty information or doing nothing with it. So notification perms would then be seen as enabled, but nothing is actually being send to the user.

Unfortunately Google isn't really exposing this to users, so you need something like App Ops or adb to set it up.

Or probably the most straightforward one, which is SSL termination. Most backend software usually has very bad support for HTTPS communication, while it's typically extensively documented for something like nginx. It also catches some other strangeness like making it easier to update the certificate.

The biggest risk is incorrect usage of the default_server directive, the proper way in which to handle it isn't usually taught in most "here's how you use nginx" tutorials. Most usually just have you edit the default server blocks.

Tldr that covers 99% of all cases: you want 2 default server blocks, one on port 80 and one on port 443. The one on port 80 should only return 444 (an internal nginx status code that stops the connection immediately with no response), while the one on port 443 should use ssl_reject_handshake to terminate the SSL connection as quickly as possible without causing strange errors (you also need a self-signed certificate because otherwise openssl refuses to do protocol negotiation correctly, but the cert doesn't actually do anything). After that, specify your actual domains as separate server blocks using server_name (including a separate one for each to do the port 80->443 redirect).

Arguably this should be the default configuration shipped by distros, but it isn't for some reason, which doesn't help matters.

If you're using nginx/apache/literally anything that does reverse proxying correctly, this shouldn't be a problem unless you're routing all traffic over default_server rules unstead of server_name (or the equivalent).

They should be stopping this attack at the door (even if only to clean out your logs from scraper door knocks), which is probably why it went unnoticed for years. I don't think anyone would be deploying {A,W}SGI servers on public facing ports these days. Even if only because SSL termination is much easier in the proxy layer.

Also good lord that ARS article is a mess. What the hell happened there? An ASGI server isn't unique to AI or anything, it's just a regular supply chain dependency. I kinda expect better from ARS on stuff like this.

In 2 years the contract is up for renegotiation to a different entity (and there's now plenty of political pressure to go with a different one), so I don't think it's a problem by then.

Tying the process up in the courts for that period is also a political victory, since by the time it'd be resolved, Solvinity wouldn't have the contract anymore anyways.

The FSFE isn't nearly as impractical as the FSF is. Unlike the FSF, they're actually getting results, typically by lobbying politicians and trying to get governments to require that the code made for them is publicly available. From everything I've seen of them, they're much more capable of meeting people where they're at; take this lawsuit as an example.

The FSF wouldn't participate in a lawsuit like this because from the FSFs ideological perspective, the mistake is allowing Apple to have a closed source system to begin with (because they declared victory in the 90s and since then have shifted towards blaming users for not using Free Software); at most you'd see a head-up-ass press release after the lawsuit is settled, because that's what the FSF usually does; probably easier than actually putting in the effort to do anything to advance Free Software politically. The FSFe from what I can read in this post is actually cognizant that Apple actually has a market share and that opening up application development on Apple devices is a major step to ensuring a healthy Free Software ecosystem.

The FSF these days is a decrepit organization whose primary purpose in practice is to enable Richard Stallman to not have to participate in modern society and to host his philosophical screeds. It's issues are so specific to it that in terms of FOSS, they're a historical artifact at most. Even in the US, the SFC does more for the average free software developer (ref. the recent Bambu incident where they stepped up to help a developer from getting legal nastygrams from the company in question.)

It's in theory already possible with iDeal from what I can tell (I've seen companies that use subscriptions set up an initial iDeal payment and then convert it into a regular recurring SEPA Direct Debit), but I'm going to assume that the process is kind of messy since I haven't seen many companies implement the system in that way.

Direct Debit is very nice, largely because your bank manages the subscription; companies have to declare the payment ahead of time and if you get balance mixed up for some reason, then the bank will just do the payment whenever your balance is correct if it happens within a week. I've had credit cards decline on subscriptions before because I didn't have enough loaded up on them. Never had that issue with SEPA.

Either that or "credit cards just work", so very few entities bothered until now.

It's a bit more loaded than that. 538 post-Nate Silver had a model setup that was apparently kind of a mess. 538 was apparently sending strange messages to Republican leaning polling agencies, demanding they gave far more detailed audit information than usual (with the implication obviously being that they were fraudulent pollsters), and the guy running the site had fairly openly tuned his model on the assumption people cared about certain talking points. 538 was predicting Biden victories even when the polls were so overwhelmingly against him that not even the most Democratic leaning polling agencies had trust in him; even if you aren't running difficult math, that means something has gone wrong with the model.

(Something which got worse after Harris was picked, although every polling aggregator went barmy - there's suspicions that a lot of polling agencies aggressively normalized their data to avoid being seen as biased, leading to an almost 50/50 split.)

npm ci does indeed prevent that. The issue isn't really with npm in specific. Rather, it's with build tools like Microsoft's Oryx, which get pushed in GitHub Actions if you're using Azure App Service. That one by default uses `npm install` on older versions (it's been changed nowadays, but Azure's generated action files have a bad habit of generating with older versions of the actions they're using), even though it's specifically meant for CI usage.

In general, use of npm ci is usually sparsely documented - most node projects you can find just recommend using npm install during the setup, suggesting a failure in promoting it's availability (I only know of it because I got frustrated that the lockfile kept clogging up git commits whenever I added dependencies with what looked like auto-generated build-time junk).

Everywhere in the sense of "I have a USB stick/SD card, what do I format it to so that every major device I'm using can read it".

In practice, every OS has its preferred system and the rest has varying levels of "I guess this works", with FAT32 and exFAT being the only real cross-platform options.

To wit:

* NTFS is only really properly and fully supported on Windows. Apple mounts it read-only. Linux can certainly mount NTFS and do some basic reads and writes. Unfortunately for whatever reason, the Linux fsck tools for NTFS are absolutely terrible, poorly designed and generally can't fix even the most basic of issues. At the same time, mount refuses to work with a partially corrupted filesystem, so if you're dealing with dirty unmounts (where the worst case usually is some unclosed file handle rather than data loss, but this also happens if you try to mount a suspended Windows parititon, which isn't uncommon since Windows hibernates by default and calls it fast boot), that's a boot to Windows just to fix it.

* Apple filesystems basically only work on apple devices. It's technically possible to mount them on Linux, but you end up digging into the guts of a bunch of stuff that Apple usually just masks for you.

* ext4 is only properly read/write under Linux and requires external drivers under Windows (which may not work properly either, as corruption issues are common).

FAT32 is reliable in that any OS can fsck/chkdsk it and properly mount it without needing special drivers, but is hindered by ancient filesize limitations. exFAT, at least for most cases, is the only filesystem you can plug into most devices and expect more or less the same capabilities as FAT32 (read/write support, can fix filesystem corruption.)

Out of the os specific ones, NTFS seems like it has the most potential to be the one filesystem that works everywhere; it's modern, works good-ish on most devices, it's just that the fsck/chkdsk tooling is awful outside of Windows.

There's a name for when a virus scanner finds a program that may have a legitimate purpose, yet is typically bundled into other software in a malicious manner.

It's called a PUP, or Potentially Unwanted Program and most anti-viruses offer to remove them. They can be legitimately installed, but often aren't. (Usually they were shipped in the installers of legitimate software downloaded from sketchy distributors.)

Random AI models being shipped with Chrome is very much a PUP. The user wanted to browse the internet, not use a model. They'd install an extension if they wanted that.

The Ask toolbar was seen as a virus. Mozilla had massive user bleed in Firefox due to installing sponsored extensions in the browser. The only reason this shit isn't regarded the same way is because it's both done by Google and because it's labeled with AI, so all AI bros have to retroactively find an excuse to justify it.

To be fair, most IDEs will usually try to commit their own workspace configurations to a git repo unless you tell them off with a .gitignore. They tend to also exclude themselves from gitignore presets for much the same reason.

VS Code is one notorious offender in that realm; it will try to commit settings.json, even if their gitignore's are set up to ignore all other cruft.

In general, the question of what should go in the source folder is a bit of a mess. Source code, README and License make enough sense, but what about files describing project governance or CI configuration logic? Or what about files that are used to make the forge you're using render the repository in a certain way (for example: bug tracker templates). Those are all cruft insofar that they have nothing to do with code, but it's generally agreed on that you're supposed to commit those, maybe in a dot-folder if necessary.

It's the typical "cart before the horse" kind of corporate tech talk. It's pretty standard if Silicon Valley wants to sell shit that nobody actually wants; they just assume that people will want it, regardless whether or not they actually want it. Most of the tech press is too obsessed with retaining their "access" to actually be critical of this sort of thing, and most of the regular press doesn't care enough to actually investigate.

We've seen this sort of song and dance before, crypto jumps to mind. Remember when social media sites suddenly were all about those hexagonal avatars? Most of this stuff is really in that same vein.

(Which to be clear, users don't want this. AI pushes by pretty much all recent user feedback metrics are largely tiring out users and reek of corporate desperation to sell shit. It's only a very specific subsection of Silicon Valley that wants to stuff AI in everything like this.)

Forge federation seems like a bad idea to me. If you want to go the route of decentralized project management (note that git as a VCS tool is already decentralized for this purpose), you're probably much better off modernizing the git-over-email workflow instead.

Decentralizing the code isn't an issue; cloning repo's between servers is so standard that any forge can import a code repo from any other forge.

The difficulty is ancillary stuff like issue trackers, wikis and MRs, but using a federated protocol for that seems ill-advised given the much weaker safeguards against spam. Mailing lists have a very large existing body of work on the matter of dealing with spam and a proven method of mirroring/archival. (Most git wikis are just git repositories with a different renderer.)

The main reason nobody likes doing git-over-email is mostly just because it's very user-unfriendly to set up (since modern mail clients typically aren't correctly configured to deal with them). It's a very developer oriented workflow in the worst way possible. A modernized mailing list program that automatically takes care of things like reformatting emails/not leaking email addresses to the general public would go a long way to make it easier to deal with.

I sorta alluded as to how the government is pushing the sale through, but to reiterate more clearly, and with political detail:

* In early 2025, the previous dutch government lost majority coalition support. The previous government remains in power until a new one is elected, but isn't expected to make major decisions any more; they're effectively just stewards to ensure the country isn't totally leaderless[0] (this is also called a demissionary government here).

* In late 2025, a new second chamber is elected and work on a coalition begins. Until a new coalition is formed, the previous government remains in power.

* In November 2025, right around this, Kyndryl announces it's takeover of Solvinity. The demissionary government gives initial approval for the takeover and decides that the takeover won't mess with the DigiD contract.

* In January 2026, the deal begins to fall under scrutiny in IT/privacy circles and some political parties express their concerns, but not much media attention is drawn to it at first. The ACM (dutch antitrust authority) also gives it's approval for the sale. All this still happens under the demissionary government.

* In February 2026, the new coalition government is sworn in. Scrutiny on the deal is starting to intensify and media coverage becomes more public.

* In late April 2026 (as in, last week), parliament passes a motion to request to change away from Solvinity in 2028. At the same time, the minister responsible for the sale is answering press questions about the sale, indicating he doesn't intend to block the sale. Just four days ago, the minister publishes a formal letter to parliament, effectively saying that they aren't stopping anything and that the government already gave preliminary approval to extend the DigiD contract with Solvinity (a separate matter, but just as related) back in March (so under the current government). They expect to ink it before the end of next week (May 6th.)

Somewhere between March and April, some internal government employees also step to the press to warn them about the sale in terms of a national security threat, but I don't exactly recall when on the timeline that happened.

The reason why the government is getting the blame for it isn't just inaction; they aren't standing by and letting something they had no involvement with (since the previous government was demissionary) happen - they're actively choosing to continue the motions of the previous demissionary government - including signing contract extensions that the previous government wasn't involved with - in spite of the very clear pushback they're getting from doing so.

[0]: This is the abbreviated version - somehow the previous government managed to lose coalition support twice, even when it was already demissionary. It's not normal for two parties to pull out like that on separate occasions.

Since a lot of this discussion is talking around the actual situation, let me try and explain it in more detail.

The dutch government has an authentication system called DigiD. It's effectively an OAuth protocol for government sites, and one of the few ways in which the Dutch government has centralized IT. Every dutch citizen can get access to it, and probably will need it at some point to deal with the government (paper options are meant to exist, but you can already guess on how easy the availability of that is.)

DigiD is currently hosted by a dutch company named Solvinity and developed by Logius (the governments in-house IT development organization). Solvinity is currently in the process of being bought out by another company, Kyndryl, which is based in the US. The government approved the takeover under the previous coalition (who are no longer in power.) The takeover currently is under extreme public scrutiny because of everything to do with the US - most people are at least vaguely aware of the deadly combination of the US CLOUD/PATRIOT laws, which would compel Kyndryl to hand over data on any dutch citizen to the US government for any reason[0]. The US government right now is not exactly behaving like a good steward with the powers it has, instead favoring maximum exploitation within (and outside, if the lawsuits are any indication) it's legal limitations, and is also verbally attacking it's own allies near constantly. Given DigiD is effectively a list of personal information on almost every dutch citizen, it's probably a bad idea to hand access to it over to a hostile foreign country.

On an employee level, the takeover is deeply unpopular - some government workers have actively reached out to the press to warn about the deal, something which very rarely happens as government workers aren't expected to publicly break with government policy. This has led to a motion in the second chamber (parliament) to change DigiDs hosting from Solvinity to another provider being passed... in 2028, for a deal set to go through in a much shorter timespan. At the same time, the government (this time: the elected politicians) is unwilling to reconsider it's stance on the Solvinity takeover, claiming that because it already said it was OK before, it can't change its mind now.

[0]: It's also, almost certainly illegal in a GDPR/AVG (local version of GDPR) sense. US/EU privacy laws are fundamentally incompatible with one another because of these two laws, and the courts keep shooting the international data transfer agreements to bits every time. Even on a basic level, having your government authentication systems legality tied to whether or not Max Schrems wins his court cases is a bad idea.

Besides Jones and his lawyer absolutely botching his defense and basically giving up the case (and pissing off the courts as I understand it, which is a bad fucking idea and usually also leads to larger fines), the $1.4 billion is just what Jones managed to rack it up to before entering bankruptcy proceedings, which froze his debt collectors out for a bit.

Alongside the class action, Jones was iirc also facing several separate lawsuits, so what you're seeing here is multiple lost lawsuits (I think he lost 4?) adding up.

The bankruptcy also doesn't wipe the slate clean for Jones afaiu, because he specifically was found to be malicious in his behavior. Court debts aren't wiped in that situation. He's still on the hook for that.

The big reason is that domestic ISPs don't want to switch (not just in the US, but everywhere really.)

Data centers and most physical devices made the jump pretty early (I don't recall a time where the VPS providers I used didn't allow for IPv6 and every device I've used has allowed IPv6 in the last 2 decades besides some retro handhelds), but domestic ISPs have been lagging behind. Mobile networks are switching en masse because of them just running into internal limits of IPv4.

Domestic ISPs don't have that pressure; unlike mobile networks (where 1 connection needing an IP = 1 device), they have an extra layer in place (1 connection needing an IP = 1 router and intranet), which significantly reduces that pressure.

The lifespan of domestic ISP provided hardware is also completely unbound by anything resembling a security patch cycle, cost amortization or value depreciation. If an ISP supplies a device, unless it fundamentally breaks to a point where it quite literally doesn't work anymore (basically hardware failure), it's going to be in place forever. It took over 10 years to kill WEP in favor of WPA on consumer grade hardware. To support IPv6, domestic ISP providers need to do a mass product recall for all their ancient tech and they don't want to do that, because there's no real pressure to do it.

IPv6 exists concurrently with IPv4, so it's easier for ISPs to make anyone wanting to host things pay extra for an IPv4 address (externalizing an ever increasing cost on sysadmins as the IP space runs out of addresses) rather than upgrade the underlying tech. The internet default for user facing stuff is still IPv4, not IPv6.

If you want to force IPv6 adoption, major sites basically need to stop routing over IPv4. Let's say Google becomes inaccessible over IPv4 - I guarantee you that within a year, ISPs will suddenly see a much greater shift towards IPv6.

Not just antivirus, there's also file locking.

Windows has a much harsher approach to file locking than Linux and backup software like BackBlaze absolutely should be making use of it (lest they back up files that are being modified while they back them up), but that also means that the software effectively has to ask the OS each time to lock the file, then release the lock when the software is done with it. With a large amount of files, that does stack up.

Linux file locking is to put it mildly, deficient. Most software doesn't even bother acquiring locks in the first place. Piling further onto that, basically nobody actually uses POSIX locks because the API has some very heavy footguns (most notably, every lock on a file is released whenever any close() for that file is called, even if another component of the same process is also having a second lock open). Most Linux file locks instead work on the honor system; you create a file called filename.lock in the same directory as the file you're working on, and then any software that detects the filename.lock file exists should stop reading the file.

Nobody using file locks is probably the bigger reason why Linux chokes less on fast iteration than Windows, given that Windows is slow with loads of files even when you aren't running a virus scanner.

I'm not saying Backblaze should adapt to git; the issue isn't application related (besides git being badly configured by default; there's a solution with git gc, it's just that git gc basically never runs).

It's that to back up a folder on a filesystem, you need to traverse that folder and check every file in that folder to see if it's changed. Most filesystem tools usually assume a fairly low file count for these operations.

Git, rather unusually, tends to produce a lot of files in regular use; before packing, every commit/object/branch is simply stored as a file on the filesystem (branches only as pointers). Packing fixes that by compressing commit and object files together, but it's not done by default (only after an initial clone or when the garbage collector runs). Iterating over a .git folder can take a lot of time in a place that's typically not very well optimized (since most "normal" people don't have thousands of tiny files in their folders that contain sprawled out application state.)

The correct solution here is either for git to change, or for Backblaze to implement better iteration logic (which will probably require special handling for git..., so it'd be more "correct" to fix up git, since Backblaze's tools aren't the only ones with this problem.)

I decided to look into this (git gc should also be doing this), and I think I figured out why it's such a consistent issue with git in particular. Running git gc does properly pack objects together and reduce inode count to something much more manageable.

It's the same reason why the postgres autovacuum daemon tends to be borderline useless unless you retune it[0]: the defaults are barmy. git gc only runs if there's 6700 loose unpacked objects[1]. Most typical filesystem tools tend to start balking at traversing ~1000 files in a structure (depends a bit on the filesystem/OS as well, Windows tends to get slower a good bit earlier than Linux).

To fix it, running

git config --global gc.auto 1000

should retune it and any subsequent commit to your repo's will trigger garbage collection properly when there's around 1000 loose files. Pack file management seems to be properly tuned by default; at more than 50 packs, gc will repack into a larger pack.

[0]: For anyone curious, the default postgres autovacuum setting runs only when 10% of the table consists of dead tuples (roughly: deleted+every revision of an updated row). If you're working with a beefy table, you're never hitting 10%. Either tune it down or create an external cronjob to run vacuum analyze more frequently on the tables you need to keep speedy. I'm pretty sure the defaults are tuned solely to ensure that Postgres' internal tables are fast, since those seem to only have active rows to a point where it'd warrant autovacuum.

[1] https://git-scm.com/docs/git-gc

I can understand in theory why they wouldn't want to back up .git folders as-is. Git has a serious object count bloat problem if you have any repository with a good amount of commit history, which causes a lot of unnecessary overhead in just scanning the folder for files alone.

I don't quite understand why it's still like this; it's probably the biggest reason why git tends to play poorly with a lot of filesystem tools (not just backups). If it'd been something like an SQLite database instead (just an example really), you wouldn't get so much unnecessary inode bloat.

At the same time Backblaze is a backup solution. The need to back up everything is sort of baked in there. They promise to be the third backup solution in a three layer strategy (backup directly connected, backup in home, backup external), and that third one is probably the single most important one of them all since it's the one you're going to be touching the least in an ideal scenario. They really can't be excluding any files whatsoever.

The cloud service exclusion is similarly bad, although much worse. Imagine getting hit by a cryptoworm. Your cloud storage tool is dutifully going to sync everything encrypted, junking up your entire storage across devices and because restoring old versions is both ass and near impossible at scale, you need an actual backup solution for that situation. Backblaze excluding files in those folders feels like a complete misunderstanding of what their purpose should be.