xurukefi

How do clients get the address for a nice domain name without the addresses in public DNS?

They don’t. You put the addresses in public DNS.

Or...

- Don't use split DNS. Don't use any special internal or dev domain. Leave it to your infrastructure to route/NAT those public IPs to your internal network.

- Don't use the HTTP-01 challenge. Use DNS-01.

- Don't run your own internal CA. Use Let's Encrypt. If you care about name leakage (CT Logs), use wildcard certs. Use a central reverse proxy/load balancer for termination.

Sure, but maybe there are other ways to control Googlebot in a similar fashion. Maybe even with a pristine looking User-Agent header.

That's actually a really neat idea.

There are ways to work around this. I've just tested this: I've used the URL inspection tool of Google Search Console to fetch a URL from my website, which I've configured to redirect to a paywalled news article. Turns out the crawler follows that redirect and gives me the full source code of the redirected web site, without any paywall.

That's maybe a bit insane to automate at the scale of archive.today, but I figure they do something along the lines of this. It's a perfect imitation of Googlebot because it is literally Googlebot.

But it is reliable in the sense that if it works for a site, then it usually never fails.

Exactly. If I was an admin of a popular news website I would try to archive some articles and look at the access logs in the backend. This cannot be too hard to figure out.

Because it works too reliably. Imagine what that would entail. Managing thousands of accounts. You would need to ensure to strip the account details form archived peages perfectly. Every time the website changes its code even slightly you are at risk of losing one of your accounts. It would constantly break and would be an absolute nightmare to maintain. I've personally never encountered such a failure on a paywalled news article. archive.today managed to give me a non-paywalled clean version every single time.

Maybe they use accounts for some special sites. But there is definetly some automated generic magic happening that manages to bypass paywalls of news outlets. Probably something Googlebot related, because those websites usually give Google their news pages without a paywall, probably for SEO reasons.

Kinda off-topic, but has anyone figured out how archive.today manages to bypass paywalls so reliably? I've seen people claiming that they have a bunch of paid accounts that they use to fetch the pages, which is, of course, ridiculous. I figured that they have found an (automated) way to imitate Googlebot really well.

I hate NAT with a passion. It's a terrible technology, whose disruptive nature has probably prevented any novelty on the transport layer. But this article is oversimplifying things.

It is well known that NAT is not meant for security and that NAT is not a firewall. But one cannot deny that it implicitly brings some "default" security to the table. With NAT it's basically impossible to screw you over because there is no meaningful practical way to allow inbound connections without the client explicitly defining them (port forwarding). With IPv6, you could have a lazy vendor that does not do any firewalling or a has a default allow policy or maybe buggy firewall. With NAT that is not possible. There is no lazy/buggy NAT implementation that allows inbound connections for your entire network, because it is technically not possible. When a NATting device receives a packet with a destination port that has not previously been opened by a client, it does not decide to drop this packet because of a decision by the vendor. It drops the packet because there is simply no other option due to the nature of NAT. That is what people mean when they talk about the inherent "security" of NAT.

Again, NAT is terrible. We need to finally get rid globally of IPv4 and all the NATting that comes with it. But let's keep it to the facts.

For me, type hints are mainly useful because they're the only reliable way to get decent IDE auto-completion. Beyond that, they feel like a bolted-on compromise that goes against the spirit of Python. If you really need strict typing, you're probably better off using a statically typed language.

The LaTeX community is astonishingly good at gatekeeping. I can't think of another field where the adoption of a clearly superior modern alternative has been so slow. For some reason, they seem to take pride in clinging to a 50-year-old typesetting system—with its bloated footprint, sluggish compilation, incomprehensible error messages, and a baroque syntax that nobody truly understands. People have simply learned just enough to make it work, and now they treat that fragile familiarity as a virtue.

removed

The question then becomes, whether Adblockers could use this information to skip the ads. It's a cat and mouse game.

I wouldn't call it a cat and mouse game because there is nothing from a technical point of view that prevents adblockers to use this information to skip ads. Unless YouTube gets completely rid of the concept of timestamps for their videos, they will always lose this battle.

For one thing, this approach seems to inherently conflict with the fact that you can link directly to a particular timestamp in a YouTube video, either in an external link using the `&t=...` URL parameter, or by just including a timestamp in a YouTube comment.

The more I think about this, the more I belive that this is literally the only reason that ad blocking cannot be meaningfully defeated for video on emand. Because of the concept of referencing a fixed point in the video by a time stamp, there will always need to be a mechanism to offset the time stamp with respect to the injected ads, which, in turn, gives ad blockers the ability to find out where the ads are exactly.

What does "server side injection" actually mean?

The way ads usually work is that they are separate video files that are fetched by the YouTube client (e.g., the browser) and then displayed to the user. Ad blockers modify the content of the web site such that the URLs to those ads (usually embedded in some JSON object from an API endpoint or something like that) get removed so that no ads are displayed.

Server side injection in this context means that the server renders a video file on demand that contains the original clean video plus a few ads here or there. Blocking the ads now is much harder because you cannot simply manipulate API responses containing refrences to those ads because there is only this one video file. Instead you would need to implement a mechanism that skips those ads in the player.

AFAIK server sie injection is already done on twitch for live streams where blocking ads is really basically impossible because you cannot skip anything in a live stream. I think the best solution for twitch to get rid of ads is to use a VPN/Proxy in a country where no ads are delivered for contractual reasons.

It's a nice idea, but I don't think it adds enough clarity to the code to justify the messy compiler warnings and errors that this kind of preprocessor abuse will eventually cause.

With PlayReady, as with any other DRM scheme really, there are different tiers. There is SL2000, which is done completely in software (whitebox crypto), and there is SL3000, which does require a TEE. Which tier is requried for which type of content is driven by streaming provider or studio requirements. I think it is pretty common to allow content up to 1080p to be used with whitebox crypto, whereas 4k+ content will require hardware DRM.

The "client" whose "identity" is abused here is not an end user. A "client" in this context is a program or library that talks to the license servers and receives the content decryption keys. On my Windows machine I see a "Windows.Media.Protection.PlayReady.dll", which I guess is the client that they cracked. Maybe there are also other clients that are widely accepted by license servers.

The attack essentially means that they could write a program themselves that acts as "Windows.Media.Protection.PlayReady.dll" to get decryption keys from a server. What will happen now is that Microsoft will deprecate the client and release a new one with new obfuscation and new keys. The license servers will start rejecting the old cracked client. And then people will crack the new client. And the cycle continues.

Reading the comments here makes me feel guilty. I'm sitting on probably a few hundred files and folders called something like tmp, tmp1, foo, foo23, foobar, testxyz, etc... They all hold probably very irrelevant stuff and are safe to delete, and I have yet to resort to those files for rescue, but you never know! Every now and then I collect them and put them in an archive folder. I'm now at "archive10".

Since I'm a bit late to the party and feeling somewhat overwhelmed by the multitude of articles floating around, I wonder: Has there been any detailed analysis of the actual injected object file? Thus far, I haven't come across any, which strikes me as rather peculiar given that it's been a few days.

Thanks for the insight. The fact that Paper Mario uses optimizaion flags makes this project even more fascinating. Great work.

Fascinating work. I know that people did the same for Super Mario 64 [1]. It is still unbelievable to me that they can generate a bit-by-bit identical copy of the original ROM by simply running some old gcc on actual C source code files. IIRC the main insight was that Nintendo did not use any optimization flags which made it possible to create a 100% matching binary. I've never really looked into it, but I guess the did some try and error to figure out the exact gcc version that Nintendo used to make sure that they get 100% identical binary code?

[1] https://github.com/n64decomp/sm64

While that sounds reasonable, is there any evidence that this is the actual reason? Widevine is also used by other ubiquitous services such as Netflix, Prime, Hulu, etc.. and yet WV still remains an insurmountable barrier for most people, even in its weakest form (L3).

The "xor rax, rax" that I just saw at a quick glance makes me flinch. Still putting it on my reading list though. Sounds like a really interesting little toy project.

    $ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos'
    [...]
      File "/app/yt_fts.py", line 176, in get_channel_id
        channel_id = re.search('channelId":"(.{24})"', html).group(1)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'group'

    $ python yt_fts.py download 'https://www.youtube.com/@ycombinator/videos' --channel-id UCj089h5WsDdh1q8t54K3ZCw
    [...]
      File "/app/yt_fts.py", line 191, in get_channel_name
        data = json.loads(script.string)
                          ^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'string'

It’s absolutely impossible to use any sort of adblocker on twitch.

I have never seen any ads on twitch. I just checked with DevTools and it seems like uBlock Origin simply blocks a script called "v6s.js", that is apparantly responsible for serving ads.

It will be interesting to see how this plays out. Ad blocker detection is an arms race and considering how big YouTube is, people will definitely fight back and go very far out of their way to avoid ads. This could get really ugly given that Google controls Chrome and its extension ecosystem.

Don't we already implicitly have that sort of trust at the moment with TLS certs considering that proof of ownership via DNS is quite common? Actually, any domain-based validationi, i.e., also HTTP-01, is going to be flawed if you don't trust the registries.

How about something very pragmatic: Use the same key you use for your webserver, which already has a valid SSL cert. Clients first connect to port 443 to get the public key and can verify the certificate. During the SSH handshake, the client then compares the host key with the previously obtained key of the certificate.