unqueued

The alternative is that people complain "netflix doesn't work on firefox", switch to chrome instead, which is even worse.

So what if users complain? How is it better for Firefox to do something bad just because Google is doing something bad?

Firefox is supposed to provide an alternative to what's out there. Firefox also didn't support some popular proprietary Internet Explorer features, and they never attempted to. For a time, much of MySpace didn't work as well in Firefox. But I'm glad that Firefox didn't cave, even if some users complained that they couldn't make the scrollbars neon green or make music autoplay.

Not letting Microsoft or Google dictate how they implement a web browser worked out really well for them. Chasing proprietary platforms has ruined them.

At the time EME was adopted Firefox was much more popular, I think 20% back in 2012. Video platforms were using Silverlight for DRM. There's a good chance that EME would not have gotten off the ground if Mozilla didn't embrace it, or at least not as quickly.

Mozilla should have taken a stand and refused to support EME when they had the chance. They would be better off than they are now. And there's a good chance Netflix would not have thrown away a double digit percentage of subscribers.

Instead they embraced DRM and now they have nothing.

I wish more were done to push back against this consolidation of power by these platforms.

We're like frogs that have been stoking the fuel of our own pot.

It used to be that DRM was considered to be in conflict with the browser, because it was not acting on behalf of the user. If you must have DRM, then it is on the platform to shoehorn it in through an external plugin, like Silverlight.

When Firefox adopted EME extensions, I knew it was the beginning of the end; they were rolling out the red carpet for DRM. If we make DRM a switch that can simply be thrown, then it will become the norm, not the exception. And there have been proposals for years to DRM fonts and other absurdities. If a company insists on using DRM, then they should have to shoulder the burden of doing something that a browser was never support to support.

The nightmare that we're racing toward is you will only be permitted to cache a trickle of video at a time and your TPM attestation hardware must include a token in every HTTP request. Your browser will just be a software cablebox.

They aren't happy about URLs either and would love to require that if you want to share a reference to something, you have to do it on their terms, like generating a url in their app with a hash that expires and limited in how many times it can be viewed. I'm sure influencers will still have the privilege of unlimited sharing.

They've been slowly rolling this infrastructure out for the last decade. These are not isolated inconveniences, these are coffin nails.

I think symlinks are really underappreciated for organizing files.

One cool thing you can do is to treat directories as tags when you symlink into them, since a file can 'belong' to multiple directories.

If you're using Windows, you can just use WSL and then you don't need administrative access as the OP's program does.

Here are some simple tools to let you use symlinks as tags:

https://github.com/perses76/refmat-symlink

https://mikeknowl.es/projects/ztag/

https://github.com/michaelknowles/ztag

But if anyone is curious about ways to better organize their files in general, I would check the writings of Karl Voit:

https://archive.baty.net/2018/using-karl-voits-file-naming-s...

https://karl-voit.at/2022/01/29/How-to-Use-Tags/

Simple and portable self describing filenames has served me really well for a long time across many platforms.

This script has done a lot of lifting for me and it will work with WSL with little effort:

https://github.com/mdom/squaretag

If you are iterating over a lot of files, a read while loop can be a major bottleneck. As long as you use the null options from find and pipe into xargs, you should be safe with any filename.

I've found it can reduce minutes down to seconds for large operations.

If you have to process a large number of files, you can let xargs minimize the number of times a program is run, instead of running it once per file.

Something like:

  # Set the setgid bit for owner and group of all folders
  find . -type d -print0 | xargs -0 chmod g+s

  # Make the targets of symlinks immutable
  find . -type l -print 0 | xargs -0 readlink -z | xargs -0 chattr +i

There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1].

If you have to output in a shellscript in this format, use printf %q

from man printf:

       %q     ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable
              characters with the proposed POSIX $'' syntax.

$ touch $'\nHello\tWorld\n' $ ls

One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset.

One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though.

Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in.

Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files

It is amazing how many things it breaks.

If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering.

[1] https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti...

[2] https://dev.yorhel.nl/ncdu/jsonfmt

Both Finder and explorer.exe will mung mtimes from casual browsing, usually due to metadata updates. It is a common enough problem that there are utilities to bind mtimes to files.

IMO mtime the de-facto file timetsamp metadata, since it is most widely supported. The other file timestamp metadata I found were not as useful or portable.

And people use git for storing more than just source code.

I personally find it very useful to be able to clone my podcast index and have accurate timestamps not just from the commit, but from the file itself.

Timestamps can be important for a collections of files, but they are fragile and often an afterthought.

Sometimes search indexing and build tools get it wrong, so if you know what you're doing, then tools like this allow you to provide an override.

Just browsing an old photo collection without making any changes can cause a 15 year old file to show up as a "new file", even though it's contents have not changed. This is especially true with media files that the os might take it upon itself to regenerate metadata for.

Or you may wish to store an accurate snapshot of the timestamps of cached or intermediate build artifacts.

Thanks for sharing, I love stuff like this. I get frustrated at mtimes being munged. MacOS finder is terrible about this.

If anyone is interested, I wrote a more generalized tool for storing metadata called storetouch[1] which takes a snapshot of file's mtimes, and stores them in a nicely formatted file for convenient restoration. You can run it's output by itself with bash, or easily parse it's output.

I found that the commit or author date's did not always make sense to represent the true age of a file. Especially for non-source-code documents. For example, a repo of scanned documents. Or a git-annex repo of podcasts going back 20 years.

Instead, I treat the commit date as the commit date, and I have a versioned sidecar that stores the actual original mtimes of the files.

It can also handle all valid POSIX filenames (a surprising amount of utilities don't, including tools like git-restore-mtime, last time I checked). If you're already using a tool like git-annex or datalad though, they have the ability to store metadata for each file as well.

I would also suggest checking out git-store-meta[2] which was the original inspiration. It stores more than just mtime, but it is more closely tied to git.

[1] https://github.com/unqueued/storetouch

[2] https://github.com/danny0838/git-store-meta

You can already use it with git-annex to store binaries using the git-annex-remote-git-aafs[1] special remote.

Although I would be careful and make sure you understand what it is doing to your branch namespace. Even though in the worst case it would not save any space over directly committing binaries, they are in orphan branches that can be pruned without rewriting history.

But even so, you can just use any number of git-annex special remotes to bypass using git for sharing files.

They may eventually add first-party support for git-annex. But nothing is stopping you from using it now.

[1] https://github.com/domo141/git-annex-remote-git-aafs

I think "confabulation" is a way more accurate term, I wish it had stuck instead of "hallucination".

A hallucination is a problem with input. Confabulation is false output.

Confabulation is when a person mistakenly recalls details and tries to "fill in the blanks", without realizing what they are saying is untrue.

That's what's great about btrfs. I started using it that way too. It was nice to be able to take instant snapshots of my fs. I just amended my existing backup scripts to act on temporary snapshots instead. Then, I started dividing things up into subvolumes.

I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.

I also highly suggest reflinks. Once you start using them you will use them all the time.

I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.

I will probably give bcachefs on one of my machines.

I started using btrfs back when it was way less stable, but that instability was more than offset by the other features.

The last time I lost a btrfs filesystem was four years ago when it was less stable, but it did not matter because it only amounted to about an hour of logs lost, and I reconstructed it in minutes. So btrfs is a must on any embedded devices. It frees me to use cheaper media, because I can have two copies of all data and metadata, and compression, and fine grained external snapshots.

Zfs is great for making big complex arrays and if you plan out how your are storing your data ahead of time. But it has a steeper learning curve and is more seperate from the underlying OS.

Btrfs fills this perfect niche for me of being a drop in replacement for ext4, and being really flexible and offering some of the nicer features of zfs. (differential snapshots, checksumming, compression)

So I'm excited to try bachefs as an alternative to btrfs.

This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots.

I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.

I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.

I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.

I actually have something I've been meaning to put on github. I wrote it over a decade ago and I need to dockerize it. I only made it in Vagrant.

But it can do something iVentoy can not.

https://www.iventoy.com/en/faq.html:

   The computer which run iVentoy must be in the same LAN with them.
   Besides, there must no other DHCP servers in the same LAN.

It turns out that multiple DHCP servers can co-exist on the same network segment. So I just have a second dhcp server solely for a pxe bootstrapper. At the time, at least the initial download must be tftp, but I used a stripped down ipxe stub that took advantage of the universal undi network drivers, and then loaded the rest (usually syslinux) over a fast HTTPS connection.

From there, iSCSI was no problem, and I could also use tools like plop, memdisk, or grub4dos, as well as Windows 7/8/10 with wimboot.

I remember for some distros like Knoppix, we would use an NBI driver to just mount the iso directly over the network, which could be far simpler and more performant than NFS.

EDIT: To be honest, it is simple enough that it doesn't have to use Docker. It is mostly dnsmasq and PHP.

My biggest mistake after being laid off was to not be discerning enough about my next job. You need to be willing to say no to a crummy job offer, even as the bills pile up.

John, at around that time you mentioned, you gifted me with my first shell account on one of your coffeshop servers, Reva. I just wanted to say THANK YOU. Up until then, I had read books about unix from the library, and had failed to install Mandrake on my laptop, but being able to access such a well configured NetBSD environment was really helpful and set me on a good path.

I had never been truant from school before up to that point, but sometimes life rewards you for taking risks.

I was hoping that using hardware keys would eliminate some of the security hoops that we have to jump through. And it does seem to help. But the whole reason that I have a key is so I do not have to supply my phone number, and I have a more trusted way of proving my identity, even if I am connecting from an unusual location.

This makes me worried, because I am pretty sure Google is going to start removing keys based on attestation certificates.

I believe that this is much more about rate limiting than about security for the end users.

No, that is not correct, git-annex uses a variety of special remotes[2], some of which support deduplication. Mentioned in another comment[1]

When you have checked something out and fetched it, then it consumes space on disk, but that is true with git-lfs, and most other tools like it. It does NOT consume any space in any git object files.

I regularly use a git-annex repo that contains about 60G of files, which I can use with github or any git host, and uses about 6G in its annex, and 1M in the actual git repo itself. I chain git-annex to an internal .bup repo, so I can keep track of the location, and benefit from dedup.

I honestly have not found anything that comes close to the versatility of git-annex.

[1] https://news.ycombinator.com/item?id=33976418

[2] https://git-annex.branchable.com/special_remotes/

I have a 1.96 TB git repo: https://github.com/unqueued/repo.macintoshgarden.org-fileset (It is a mirror of a Macintosh abandoneware site)

  git annex info .

And the datalad project has neuro imaging repos that are tens of TB in size.

Consider whether you actually need to track differences in all of your files. Honestly git-annex is one of the most powerful tools I have ever used. You can use git for tracking changes in text, but use a different system for tracking binaries.

I love how satisfying it is to be able to store the index for hundreds of gigs of files on a floppy disk if I wanted.

For bigger annex repos with lots of pointer files, I just disable the git-annex smudge filters. Consider whether smudge filters are requirement, or a convenience. The smudge filter interface does not scale that well at all.

I combine git-annex with the bup special remote[1], which lets me still externalize big files, while benefiting from block level deduplication. Or depending on your needs, you can just use a tool like bup[2] or borg directly. Bup actually uses the git pack file format and git metadata.

I actually wrote a script which I'm happy to share, that makes this much easier, and even lets you mount your bup repo over .git/annex/objects for direct access.

[1] https://git-annex.branchable.com/walkthrough/using_bup/

[2] https://github.com/bup/bup

I never use split-screen view for Markdown. Split-screen makes sense for something more abstract like latex or html. But I do not see the benefit for Markdown. I would much rather have 50% of my screen.

I was astonished when I noticed requests like this after I logged into a customer portal:

  curl 'https://hondalink.honda.com/api/VehicleFeatures/[VIN NUMBER]' -H 'hondaHeaderType.country_code: US' -H 'hondaHeaderType.language_code: en' -H 'Authorization: Basic [DIGEST]' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Car-Type: HONDA' 

They clearly intended to impelement authentication on their API endpoints, but didn't finish it, becuase the client is sending an Authorization header. But I don't know if their servers are even configured to check them. And they should certainly use JWT instead of digest.

This is a really cool product! I'm so glad someone is finally trying to solve this problem.

Not having control of your secrets is a big problem. It is why I continue to use TOTP for some things.

Maybe lean into the practical problem this solves and how it empowers the user?

I am going to suggest this to people. But I feel like I'd have to explain to them what it does, and why it is cool. I remember a few years ago when Keybase took off a few years ago. I think Keybase did a pretty good job of presenting itself to it's audience.

I know plenty of people who understand what a Yubikey is, but might not understand what this app is.

Personally, I love that I could keep backups of my TOTP secret keys in encrypted qr codes on paper in a safe, if I wanted to.

I think it is deeper than just the technical part of it. I think people really like anchoring their accounts to something that they can "see".

It would be really easy to let you import and export your secrets to physical media. And there are some interesting options with chaining your secrets to other hardware tokens (that you are in control of).

Of course it gets very tricky when suggesting anything security related. I don't want to tell people that they shouldn't use hardware tokens. But I think that relying on hardware tokens can be really inaccessible, which can itself be make a system less secure.

Best of luck!

I added a PR to fix rsync, mosh, and some other tools, since they will be broken out of the box.

But it seems this project has fallen behind on PRs.

If you would like to have this fix, you can do this after cloning the repo:

  git clone https://github.com/mathiasbynens/dotfiles && cd dotfiles

  git fetch origin "+refs/pull/*/head:refs/remotes/origin/pr/*" && git cherry-pick origin/pr/963

I was amazed when I learned that most powerbanks do not function that way. Some of them will switch modes if you unplug them, but there is still an interruption.

I was able to find a device that actually functions as a USB UPS: https://www.amazon.com/gp/product/B07ZS3WYZY/

It is so nice to be able to unplug my Pi to move it, and have it stay on. I also use them for my usb powered wifi security cameras. So far they have not failed me.

No, with JWT parsing it doesn't even have to look anything up, cached or otherwise. The JWT itself contains the information directly.

I could have a server that doesn't have any access to the database at all, and it could still render content for the user.

And if the server is for something critical, like purchases, it can still use redis on each request to verify that a JWT is still valid.

Even when storing the cookies in a database like Postgres, they are fast enough because it's obviously indexed, and it's probably cached in many usage patterns.

If you scale up high enough, it does matter.

But beyond that, it is also about decoupling the server serving the content and the user database.

To give you an example, using a traditional session cookie model, whenever my server gets a request, it has to look up the associated session info, and then possibly join multiple tables to see if a specific user for that specific session is authorized to access that content.

There are going to be many joins involved. Even with good caching, it will still be a more complex operation.

Even if not computationally complex, it will be logically complex.

But when using a JWT, my server actually does not need ANY db access. It merely needs to verify the cryptographic validity of the included JWT header, which uses no IO. After that, it can safely trust any user metadata included in the JWT.

So, if a user has paid for access to all premium content after 2018, you would just include a field for that in your JWT payload. Done.

It depends of course on how your business data models work of course. But you can often put enough information in your JWT to cut out a lot of account processing logic.

JWT is solving a non-problem for most people.

I think people are cargo culting JWTs.

Well, with the hub cli, it is just using GitHub's public API. I don't think it is doing anything proprietary.