jcalvinowens

One thing that concerns me is the possibility of on-die ECC interacting with ECC on the motherboard and reducing it’s effectiveness

DDR5 on-die ECC detects and corrects one-bit errors. It cannot detect two-bit errors, so it will miscorrect some of them into three-bit errors. My understanding is that the on-die error correction scheme is specifically specially designed such that the resulting three-bit errors are mathematically guaranteed to be detected as uncorrectable two-bit errors by a standard full system-level ECC running on top of the on-die ECC. But I've never found a real authoritative reference that directly says that.

HDCP1 is completely broken, you can buy HDCP2.x strippers on amazon (they're often marketed as "converters").

+1 for btrfs

If a NAS has a 1G or even 2.5G NIC, improving the filesystem performance is a waste of time... the network is the bottleneck. My N100 two-disk btrfs-raid1 NAS hits line rate on a 2.5G NIC, that's all that matters to me.

It's actually faster than I remembered:

  {0}[calvinow@sousa ~/git/linux] git describe
  v7.1
  {0}[calvinow@sousa ~/git/linux] git clean -dffxq
  {0}[calvinow@sousa ~/git/linux] zcat /proc/config.gz > .config
  {0}[calvinow@sousa ~/git/linux] time make -skj32 tar-pkg
  './System.map' -> 'tar-install/boot/System.map-7.1.0'
  '.config' -> 'tar-install/boot/config-7.1.0'
  './vmlinux' -> 'tar-install/boot/vmlinux-7.1.0'
  'arch/x86/boot/bzImage' -> 'tar-install/boot/vmlinuz-7.1.0'
  
  real    0m56.539s
  user    18m41.863s
  sys     2m8.754s

No, 90 seconds is the clean build time without ccache.

The build takes about 30-45 minutes

If you don't actually need all the drivers, you can use "make localmodconfig" to substantially reduce that. My local kernels build in 90 seconds on a 32-thread desktop machine :)

The kernel is a lot more stable than people think: I run the daily linux-next on my Debian stable gaming PC to look for bugs, and I don't find very many.

I love keyd, it uses uinput so it works on the vtty too.

https://arxiv.org/pdf/2411.09241

XFS does use /lost+found, it calls it the "orphanage directory" and xfs_repair reparents children of corrupt directories there.

Based on comments in the kernel source, it seems like the userspace fsck for JFS and F2FS will also sometimes create /lost+found. There might be more that do.

It is a weirdly common misconception that that fork() is cheap... it is O(N) on the size of the process, and it always has been.

Yes, it's copy on write... but there is a linear relationship between the size of the process and the number of page table entries required to represent it.

It's unbelievable and it's only getting worse.

A 2x32GB DDR5 kit I paid $150 for 11 months ago costs $910 today from the same retailer. A 2x16GB DRR4 kit that was $105 last year is now $230.

The RAM alone in my newest machine would sell for over double what I paid for the entire machine one year ago.

There's a tradeoff here people often ignore: how much does the power consumption increase? In my experience, you end up using more total power because the SIMD instructions are so much more power hungry.

In the cloud, you don't pay the power bill and you have no reason to care. But it's not always like that.

The waymos are so consistently badly overpriced I've stopped even bothering to look. Nobody I know rides them! But they have riders more than half the time I walk by them, so clearly they're making money off somebody...

Yeah. It's becoming unbelievable how different the prevailing opinions on this site are from those of real people I know and work with. That's always been true to some extent... but good lord, it's like reading the news in a parallel universe right now.

As others have noted, this usually happens because both sides wrote data and one side didn't read it before calling close().

Here's a little reproducer: https://gist.github.com/jcalvinowens/da57edda9a01ca9f4c4088a...

    $ gcc -O2 test.c -o test
    
    $ strace -e socket,connect,write,accept,read,close ./test --rx        
    <...>                                                                           
    socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3                                                                           
    accept(3, NULL, NULL)                   = 4                                                                            
    close(3)                                = 0                                                                            
    read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    <...>
    read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 3035
    read(4, "", 4096)                       = 0
    close(4)                                = 0
    +++ exited with 0 +++

    $ strace -e socket,connect,write,accept,read,close ./test --tx
    <...>
    socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
    connect(3, {sa_family=AF_INET, sin_port=htons(31337), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
    write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 600000) = 600000
    close(3)                                = 0
    +++ exited with 0 +++

    $ gcc -O2 -DWRITE_TO_SOCKET_BEFORE_READ test.c -o test
    
    $ strace -e socket,connect,write,accept,read,close ./test --rx
    <...>
    socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
    accept(3, NULL, NULL)                   = 4
    close(3)                                = 0
    write(4, "\250\3\0\0\0\0\0\0\250\3\0\0\0\0\0\0$\0\0\0\0\0\0\0$\0\0\0\0\0\0\0"..., 4096) = 4096
    read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
    <...>
    read(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 997
    read(4, 0x7ffd45c2d3c0, 4096)           = -1 ECONNRESET (Connection reset by peer)
    <...>
    +++ exited with 1 +++
    
    $ strace -e socket,connect,write,accept,read,close ./test --tx
    <...>
    socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
    connect(3, {sa_family=AF_INET, sin_port=htons(31337), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
    write(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 600000) = 600000
    close(3) 
    +++ exited with 0 +++

If the workers weren't forked, the entire process would die to the SIGSEGV, and when it restarted the heap would be at a new address because of ASLR. This exploit couldn't work against a threaded daemon for that reason (only one guess).

In a world where they are forked, having a randomized heap base in each worker would also defeat the brute force approach. Instead of just fork(), it could execve() itself with some arguments that tell it to be a worker and where to find its brain, that effectively do an ASLR for each worker.

Does anybody else remember the Excel spreadsheet with a bunch of drop down menus that fed 1kloc of embedded visual basic to generate a C function to program the STM32 clock registers based on your selections? Top ten silliest things I've seen in my career for sure...

Related, I have a little end-to-end example of a piece of hardware with an STM32 running bare metal firmware like this: https://github.com/jcalvinowens/ledboard

I mean... you're missing the forest for the trees, but yes I meant "address space" generally not "stack" specifically. The nginx threads are forked, it would not be that terribly complex to set up a heap with a new random address base in each worker (the only real complexity is dealing with heap allocations which happened before fork()). But the stack matters too, generally moreso.

Apache used forked processes; I don't think that's unique or a particular issue.

Of course it is... in a typical threaded daemon, the threads have randomized stack addresses. Exactly as you observed, you get unlimited tries because nginx dutifully restarts the worker process with the same literal stack address every time it segfaults. I'm willing to bet the ASLR break they claim to have relies on that, but I'd be happy to be proven wrong if they publish it :)

Both major compilers yell at you for this nowadays... it's pretty unforgivable IMHO for somebody to be asking it as an exam or interview question if the right answer isn't "undefined":

    <source>:5:10: warning: multiple unsequenced modifications to 'a' [-Wunsequenced]
        5 |     a = a++ + ++a;
          |         


    <source>:5:7: warning: operation on 'a' may be undefined [-Wsequence-point]
        5 |     a = a++ + ++a;
          |     ~~^~~~~~~~~~~

I doubt it: aslr is not as easy to break on modern Linux as everyone in this thread wants to pretend it is. And anybody who actually cares so much about security that a compromised web frontend is the end of the world should be doing other things which would additionally mitigate this...

I know they claimed they can bypass it: if that's true, they should publish it. The forking nature of nginx is uniquely bizarre and vulnerable, and I strongly suspect that's the only way they're pulling it off. I feel like that's the interesting thing here, not the buffer overrun.

Sure, but I think the github README ought to make it more clear the POC as-is doesn't work against nginx on any current Linux distro.

The POC disables aslr: https://github.com/DepthFirstDisclosures/Nginx-Rift/blob/mai...

My employers have generally been fine giving me blanket permission to contribute to specific open source projects.

The framing matters: don't say "can I please do some charity work because it makes me feel good".

Say, "can I have your permission to get free rigorous review from experts in my field, and zero out all future maintenance costs for your company by contributing my fixes to the upstream open source project?"

Because that's really how it is. No employer of mine has ever said no to that. It is entirely in their interest for you to do this, you just have to help them see it.

Yet again my plans for a productive morning have been derailed by an excellent new write up from Dmitry :)

That hand rolled DAC for the touchscreen with the eight gpio lines is hilarious.

How fun! I'm glad it was interesting, please feel free to link to it :)

This is fun, thanks for sharing. I have a much more minimalist one for x86 Linux if you want to see what that looks like: https://github.com/jcalvinowens/asmhttpd

I have a self hosting Pi Zero W running Gentoo. It started as a joke, but I kept it because it's actually occasionally useful for testing new kernel releases.

I found a fun bug with it a couple years ago: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

It is still able to build software faster than it is released. It takes roughly a month to recompile the entire system :D

Arms races suck. I've managed to find a few L7 tricks to catch the residential proxies and serve them an empty 200, but there are obvious trivial workarounds on the other end and if I start talking about them in public they won't last long... I wish I could share :/

Every few weeks I run my nginx access logs through a script that uses the same textual ASN database to tally them up and spit out a summary report. There are many different sources for periodic textual ASN databases you can parse with UNIXy tools.