dbaupp

UTF-8 encodes each character into a whole number of bytes (8, 16, 24, or 32 bits), and the 10 continuation marker is only at the start of the extra continuation bytes, it is just data when that pattern occurs within a byte.

You are correct that it never occurs at the start of a byte that isn’t a continuation bytes: the first byte in each encoded code point starts with either 0 (ASCII code points) or 11 (non-ASCII).

People use stars differently.

https://arxiv.org/pdf/1811.07643 is some investigatory research describing, among other things, 4 clusters of reasons for starring: to show appreciation, bookmarking, due to usage, due to third-party recommendation.

FWIW, pex now also has options to unzip the archive to a cache directory on startup (I believe this happens by default now, but am not at a computer to confirm), to side step the zipapp limitations that you reference.

As of a few months ago, pex supports including an interpreter, either directly inline or lazily downloaded (and cached): https://docs.pex-tool.org/scie.html

https://qntm.org/abolish explores this idea in a fair amount of detail.

Semgrep also a CLI, that can run offline and without a cloud account.

At work, we use it for enforcing a bunch of custom lint rules configured as a yaml file committed directly to our repo, entirely cloud-free.

(I may be overreading your comment as suggesting that these were reasons to use ast-grep over semgrep.)

Thanks for the up-to-date info.

.NET 6

(I’m under the impression that this was released in 2021, whereas the linked Tokio post is from 2020. Hopefully that frames the Tokio post’s more accurately.)

In addition to what you say, it can also be easier for a (appropriately-skilled) human to verify a small program than to verify voluminous parsing output, plus, as you say, there's the semi-automated "verification" of a very-wrong program failing to execute.

I have a feeling we’re talking at cross purposes here: I was actively trying to write incorrect code. This post isn’t about memmap2 crate specifically at all, it just happens to be a convenient way to get the exact (“incorrect”) syscall I wanted from Rust.

I see where you’re coming from but… it feels like you’re trying to convince me of something about the post? If you feel like convincing a larger audience of the limitations of the memmap2 crate specifically, I suggest writing your own blog posts and/or getting involved with it. :)

That probably should have been the first thing to try.

The point of the post is not "how to make mmap work best with async/await" or "how to optimise mmap with async/await", but exploring the consequence of incorrect code (and thus explaining why one might need potential remedies like those). Sorry if that didn't come across!

Do you have info about current (production) implementations that increase the number of workers?

In https://tokio.rs/blog/2020-04-preemption#a-note-on-blocking (2020), there's reference to .NET doing this, and an explicit suggestion that Go, Erlang and Java do not, as well as discussion of why Tokio did not.

As the author, I don't think there's a clear definition of "blocking" in this space, other some vibes about an async task not switching back to the executor for too long, for some context-dependent definition of "too long".

It's all fuzzy and my understanding is that what one use-case considers being blocked for too long might be fine for another. For instance, a web server trying to juggle many requests might use async/await for performance and find 0.1ms of blocking too much, vs. a local app that uses async/await for its programming model might be fine with 10ms of "blocking"!

https://tokio.rs/blog/2020-04-preemption#a-note-on-blocking discusses this in more detail.

Yes, it couldn't not block, it's obvious... but I've encountered non-trivial amounts of "magical thinking" around both async/await ("go-fast juice") and mmap ("go-fast juice") separately, so the intersection surely has a bunch of magical thinking too, where people haven't taken the time to properly think through what's going on.

Hence, my investigation to try to make the "obvious" conclusion obvious to more people.

(Author here)

(Author here)

Thank you, you've expressed one of my goals with doing this sort of investigation/getting this data far better than I have. :)

It's something that's feels obvious once the dots are connected, but I was pretty sure many people wouldn't connect these dots automatically.

Yeah, of course a synchronous call that might block the thread is blocking, I agree... but, if I didn't have the context of "we're in a comment thread about a blog post about mmap", I'm pretty sure I wouldn't flag `x[i]` on `x: &[u8]` (or any other access) as a synchronous call that is worth worrying about.

Hence the discussion of subtlety in https://huonw.github.io/blog/2024/08/async-hazard-mmap/#at-a...

It's obvious when pointed out, but I don't think it's obvious without the context of "there's an mmap nearby".

(Author here, this behaviour wasn't surprising to me, but felt subtle enough to be worth investigating, and get some concrete data.)

(Author here.)

Good point. Do you think the difference will be observable? Will it be observable on an SSD (vs. HDD)?

(Author here)

The singled thread async "traditional IO" example is NOT single threaded

The threads backing the single-threaded IO are an implementation detail of fulfilling the `.read().await` calls. The key is that there's a single coordinator thread that's issuing all the work, with the user-space runtime multiplexing tasks on that thread. I thought the fact that the "start a request and come back when it is finished" behaviour happens to be implemented via user-space threads rather than kernel-level epoll (or similar) is unlikely to affect behaviour.

I considered scaling up the number of files and using a multi-threaded runtime, but I felt that'd make everything more complicated without fundamentally changing behaviour.

However, maybe my theory is incorrect, in which case someone else can do their own experiments to provide more concrete information.

(This is referenced in a footnote: https://huonw.github.io/blog/2024/08/async-hazard-mmap/#fn:t... )

The issue is comparing 8 OS threads no async to 1 thread async is fundamentally not very useful as long as you didn't pin all threads to the same physical core

The point is not to benchmark async vs. non-async, but provide a general reference point for "mmap working well" for comparison. As you suggest you agree with the "minor" issue tag, I don't think the parallelism vs. concurrency distinction matters much here... but again, definitely happy to see some concrete data that suggests otherwise!

Little o is still an asymptotic statement: it doesn’t have to apply for small n. A definition of f(n) = o(g(n)) is something like

   lim (n -> infinity) f(n)/g(n) = 0

For instance, this function is o(n), because 1e1000/n goes to 0 as n grows.

   f(n) = 10**n if n < 1000 else 1e1000

Interesting idea.

It looks like powers of larger primes are currently rendered with the exponent ring overlapping the "main" rings, e.g. 7 and 49 look the identical, while 343 has an overlapping 2 and 3 knot.

Thanks for the kind words and the insight!

Yes, the data is the bytes 00, 01, …, FF repeating, and that pattern is highly visible with a power-of-2 encodings, but not visible with other bases (for similar reasons that 0.1 as a (binary) float doesn’t behave as people expect).

The base 10 is referring to conversion of bytes into a long decimal (base 10) integer, not that it's being stored in chunks of 10 bits.

But yes, you're right, it would be reasonable to think of this as encoding the bytes in base 1000, where each "digit" just happens to be shown to humans as 3 digits.

Good catch! I should've tested. I've added a paragraph to https://huonw.github.io/blog/2024/03/qr-base10-base64/#extre... about this.

Good point! I've added an analysis of this and other bases to https://huonw.github.io/blog/2024/03/qr-base10-base64/#fn:ot...

(I've added an analysis of this and other bases to the article: https://huonw.github.io/blog/2024/03/qr-base10-base64/#fn:ot...)

I implicitly ignored encoding schemes like base 36 and 32 (and 16, referenced elsewhere in the thread) because they're not as good as the schemes referenced in the post. The best you can get that's fully URL safe with Alphanumeric is a hypothetical base 39, referenced in a footnote, and only using 39 of the 45 possible characters has 3.9% overhead (even ignoring the 50% overhead of the https://www.rfc-editor.org/rfc/rfc9285.html encoding).

I've added an analysis of many more bases to the article: https://huonw.github.io/blog/2024/03/qr-base10-base64/#fn:ot...

Hacker News strips leading digits targeted at "listicles" (e.g. "10 ways to fizz buzz" -> "Ways to fizz buzz"), so tricks are required if the digits are actually important.

Ah, it is a good point that it might be worth comparing to, but it is far worse.

Abstractly, it requires approximately log(45)/log(16) output bits per input bit, an overhead of 37%.

Making this more concrete: each input byte is encoded as two hex digits, and two hex digits have to be encoded as two Alphanumeric characters. It thus takes 11 bits in the QR code bit stream to store 8 bits of input.

Yeah, I had had the same question! One of my earlier articles experiments with this: https://huonw.github.io/blog/2021/09/qr-error-correction/

Figure 8 and its surrounding section are the undamaged case.

We get (very) close to cross-environment reproducible builds for Python with https://github.com/pantsbuild/pex (via Pants). For instance, we build Linux x86-64 artifacts that run on AWS Lambda, and can build them natively on ARM macOS. (As pointed out elsewhere, wheels are an important part of this.)

This is not raw requirements.txt, but isn’t too far off: Pants/PEX can consume one to produce a hash-pinned lock file.