jamesdutc

I can speak, read, and write Taiwanese Mandarin (which is likely relatively underrepresented in the training sets and, which is, in my practical experience, materially different in its usage.)

The authoritative answer for this question would best come from the millions (or tens of millions) of Chinese-speakers who are currently using LLMs to write software.

However, it is my suspicion that you would see no advantages using any language other than English. While there is a certain token-level density to written texts, it seems the benefits of this (and the more recent discussion around “caveman talk”) are quite limited.

Furthermore, consider that the vast majority of textbooks, technical documentation, blog posts, StackOverflow answers, &c. are originally in English. Historically, where these have been translated to Chinese, the translations have often been of very poor quality (and the terminology and phraseology is often incomprehensible unless you also understand some English.) I would suspect that this makes up the overwhelming majority of the training sets for these models.

That said, my experience using the most recent models, is that they are surprisingly language-agnostic in a way that surpasses readily-available human capability. For example, I can prompt the LLM to translate English into something that uses German grammar, Chinese vocabulary, and Japanese characters, and I'll get an output that is worse than what a human expert could do… but where am I going to find a multilingual expert?

(Of course, I have so far only ever been impressed that a model could generate an output but never impressed with the output it did generate. Everything—translations, prose, code—seems universally sloppy and bland and muddy.)

So what I would anticipate the biggest benefit for a Chinese-speaker today… is that if they are disinterested in working internationally, they have significantly less dependency on learning English.

It's also very easy to complain about how employees have résumé-focus in their approach to their work: “why should I bother to learn some internal-only tooling that I'll never use anywhere else (for a task that I don't really even care that much about…)?”

But, to borrow a line from Warren VanderBurgh's ‘Children of the Magenta’: “(in the industry) we created you like this.”

Another key flaw of precomposed automations for rigidly-defined work-flows is that they usually exist in precisely the circumstances that give rise to their own subversion. (I might even go so far as to suggest that the circumstances are the cause of both the mistake and the maladaptive behaviours that address the mistake…)

Ultimately, deep stacks of tightly-integrated components forming a precomposed automation that enacts some work-flow—“vertical integration” as the post frames it—is obvious enough that it seems every big company tries it… only to fail in basically the same ways every time.

I just could not disagree more.

This kind of rigid, singular view of operational workflows based on precomposed automations not only constantly break but also inevitably introduce huge inefficiences.

I posted a longer comment on lobste.rs: https://lobste.rs/s/azpsqe/vertical_integration_is_only_thin...

The python dict() is already high-quality and high-performance

Yes, the CPython `PyDictObject` has been subject to a lot of optimisation work, and it is both high-quality and high-performance. I should not have implied that this is not the case.

However, there's a lot of ongoing research into even further improving the performance of hash tables, and there are regular posts discussing the nature of these kinds of improvements: e.g., https://news.ycombinator.com/item?id=17176713

I have colleagues who have wanted to improve the performance of their use of `dict` (within the parts of their code that are firmly within the structural/Python domain,) who have wanted to integrate these alternate implementations. For the most part, these implementations do not guarantee “human ordering” so this means that they can provide these tools only as supplements (and not replacements) of the Python built-in `dict`.

moving your data into an environment that lets you make more assumptions and be more restrictive with your data and operate on it there. It's how numpy, polaris, and pandas work.

Yes, the idea of a heterogeneous topology of Python code, wherein “programme structuring” is done in pure Python, and “computation” is done in aggregate types that lower to C/C++/Rust/Zig (thus eliminating dynamic dispatch, ensuring contiguity, &c.) is common in Python. As you note, this is the pattern that we see with NumPy, Polars, pandas, and other tools. We might put a name to this pattern: the idea of a “restricted computation domain.” (I believe I introduced this terminology into wide-use within the Python community.)

However, not all code shows such a stark division between work that can be done at high-generality (and, correspondingly, low-performance) in pure Python and work that can be done at low-generality (but very high-performance) within a “restricted computation domain.) There are many types of problems wherein the line between these are blurred, and it is in this region where improvements to the performance of pure Python code may be desired.

explicitly randomize iteration

In fact, we see this with CPython `set`, controlled by `_Py_HashSecret`:

https://github.com/python/cpython/blob/6eb6c5dbfb528bd07d77b...

https://github.com/python/cpython/blob/6eb6c5dbfb528bd07d77b...

https://github.com/python/cpython/blob/6eb6c5dbfb528bd07d77b...

https://github.com/python/cpython/blob/6eb6c5dbfb528bd07d77b...

This post contains a key misconception about the Python builtin data structures, that may seem like sophistry but is key to understanding the semantics (and, thus, most fluent use) of these tools.

All of the Python builtin data structures are ordered.

The distinction we should make is not between ordered and unordered data structures. Instead, we should distinguish between human ordered and machine ordered data structures.

In the former, the data structure maintains an ordering that a human being can used as part of their understanding of the programme. A `list` is human-ordered (and its order typically connotes “processing” order,) a `tuple` is human-ordered (and its order typically connotes “semantic” ordering, which is why `sorted(…)` and `reversed(…)` is rarely a meaningful operation,) a `str` is human-ordered, and `int` is ordered (if we consider `int` in Python to be a container type, despite our inability to easily iterate over its contents. Whether or not `complex` is a container or not, is pushing this idea a bit too far, in part because I don't think anyone really uses `complex`, since NumPy dtype='complex128' is likely to be far more useful in circumstances where we're working within .)

In the latter, the data structure maintains an ordering that a human being cannot use as part of their understanding of a programme (usually as a consequence of a mechanism that the machine uses as part of its execution of the programme.) A `set` is machine-ordered, not unordered. If we iterate over a `set` multiple times in a row, we see the same ordering (even though we cannot predict this ordering.) In fact, the ordering of a `set` is intentionally made difficult for a human being to predict or use, by means of hash “salting”/seeding (that can only be controlled externally via, e.g., the https://docs.python.org/3.3/using/cmdline.html#envvar-PYTHON... `PYTHONHASHSEED` environment variable.)

Historically, the Python `dict` was machine ordered. If we looped over a `dict` multiple times in a row (without changes made in between,) we were guaranteed a consistent ordering. In fact, for `dict`, the guarantee of consistency in this ordering was actually useful: we were guaranteed that `[d]` and `[d.values()]` on a `dict` (with no intervening changes) would maintain the same correspondence order (thus `[zip(d, d.values())]` would match exactly to `[d.items()]`!)

When the split-table optimisation was added to Python, the Python `dict` became a very interesting structure. Note that, from a semantic perspective, there are actually two distinct uses of `dict` that we see in use: as a “structural” or as “data” entity. (Ordering is largely meaningless for the former, so we'll ignore it for this discussion.) When the split-table optimisation was added in Python, the underlying storage for the `dict` became two separate C-level blocks of contiguous memory, one of which was machine-ordered (in hash-subject-to-seeding-and-probing/perturbation order) and one of which was human-ordered (in insertion order.) (From this perspective, we could argue that a `dict` is both human and machine-ordered, though it stands to reason that the only useful artefact we see of the latter is with `__eq__` behaviour, which this article discusses. Since “human ordering” is a guarantee, it supersedes “machine ordering.”)

This is a genuine concern, since it hinders our ability to port over high-quality, high-performance hash table implementations from other languages (since these often do not preserve any human ordering.)

However, the ship has already sailed here. I think that once insertion-ordering became the standard, this creates a guarantee that we can't easily back down from.

Malaysian guy has to watch a BBC cook make rice (https://youtu.be/53me-ICi_f8?si=0AaZ82dk_AYFqJAx&t=226)

Of course, this video is just stupid accent comedy, but we should be careful not to draw too much from it. (Let's also set aside the specifics of making fried rice.) The implication of the section of the clip you linked is that the presenter (Hersha Patel) does not know how to make rice properly, and this is evidenced by her cooking it in too much water and draining it.

But this is not correct.

There are, in fact, many different varieties of rice, different cuisines that incorporate rice as a major component, and different styles of cooking rice. Cooking (certain varieties of long-grained? rice) in an open vessel, cooking with an excess of water, and draining the water afterwards is an extremely common and popular way to prepare it for use in some cuisines: e.g., https://youtu.be/TARO_R4cE24?t=420

When this video first made the rounds some years ago, it was surprising to see how confidently people would weigh-in on this topic, despite demonstrating very little background or knowledge. (There's a big difference between saying “that's not the appropriate way to do this in this circumstance” and “that's completely wrong,” and the former creates space to derive knowledge. After all, the dish in the video is a popular one, even in cultures that predominantly eat jasmine or basmati rice, and there are interesting variations in technique and flavour that arise as a consequence!)

Mexican moms react to Rachael Ray trying to cook (https://www.youtube.com/watch?v=zFN2g1FBgVA)

I similarly do not understand why these kind of reaction videos are popular. There are slightly better versions of this format (e.g., https://youtu.be/DsyfYJ5Ou3g?t=182) but they are drowned out by this kind of fluff. What does one really gain from interacting with such criticism?

Perhaps there is something to be learnt from these situations: ones where, equipped with just a little bit of knowledge, we derive unearned confidence, and use this confidence not to venture forth more boldly in search of knowledge, but to convince ourselves of our own superiority.

Congratulations to the creators for successfully releasing this product!

(I'm trying to start with a positive tone, since I have only negative things to say about the site itself. I want to make sure that I'm coming across a critical without coming across as mean.)

I spent a few minutes generating a couple of sample stories using their prompts for the pair that I'm most qualified to evaluate “English”→“Chinese (Traditional)” and just wasn't very impressed. Honestly, I think the approach is largely a dead-end.

Let's set aside that “Chinese (Traditional)” is not a language, and that someone with experience learning or teaching Chinese ought to know this (and, as I will argue, knowing this is critical to producing high-quality educational materials!) That the creators of this tool aren't particularly familiar with the languages themselves is probably much less consequential than that they don't really appear to be familiar with the pedagogy of teaching or learning languages.

One would anticipate that the languages that most learners want to learn are subject to broad market forces, and that, as a consequence, these languages already have a variety of high-quality, human-written primary texts and educational texts (many of which may even be free-to-access!) For the language pair I tested, this is definitely true, and I would encourage every learner to start with those materials (and to avoid anything AI-generated.)

(Of course, if I wanted to learn a less-common language where materials are hard to find this might be marginally useful—e.g., Telugu probably has more total speakers than Italian, but my local high school probably has an Italian class—but I would wonder whether the training set would be good enough to accurately reproduce the language. I suppose if I wanted to learn an endangered language, where they may simply not be enough native speakers to maintain a rich catalogue of written language, then someone could train an AI to reproduce this language to aid in learning, but a similar question arises as to whether this kind of preservation or reconstruction is sufficiently “faithful.”)

It's absolutely the case that AI tools are at a point where (for common languages) they are able to reliably generate grammatically accurate language, independent of its factual accuracy. Indeed, while I could spot fluency issues in the sample stories I reviewed (since, of course, “Chinese (Traditional)” is not a language,) I could not spot outright grammatical errors. (This is an impressive accomplishment for AI models!)

But this is really a solution looking for a problem (and, in my opinion, finding the most obvious but also least useful.)

Contrast these randomly generated story with the equivalent from a human-generated educational resource. In the case of a human-generated educational resource, the quality of language may actually be worse than than that in the AI generated resource (even in the face of sloppy AI writing tends to be!) In fact, in the case of Chinese (“Traditional” or otherwise,) this is absolutely guaranteed to be the case for an introductory text. Almost all introductory texts will be written in a very choppy, repetitive style: e.g., 「那隻狗很可愛。我養的狗也很可愛。」

(It's likely the case that even intermediate and advanced learning materials will not resemble actual primary texts. e.g., I was reading the news the other day and came across the sentence 「北捷重申，無論任何年齡，各車站閘門前的黃色標線內一律禁止喝水等飲食行為，除非是身體不適或母乳哺育」 which is perfectly appropriate for an intermediate learner… except 「閘門」 is simply not useful or appropriate textbook vocabulary!)

So why is the human-generated educational material better? Well, there's a lot of design to writing these kinds of materials. How do we teach and reïterate the most broadly useful grammatical structures and vocabulary? How do we teach this in a way that maximises retention? (And, often, how do we expose the learner to useful cultural background that will help them when they visit a region where the language is spoken?)

All of this is visible in human-generated materials, yet none of this is evident in these AI-generated materials. It is, in fact, this design that makes these materials useful in the first place. In the absence of it, we end up with vocabulary lists that define 「狗：dog」 next to 「呈現：to emerge」 where a human educator would align the difficulty of these terms to the order and process in which a human learner would learn them. Similarly, a human educator knows how to evolve a student's fluency with language and understanding of tone and register, taking them from 「媽媽: mother」 to 「母親: mother」 perhaps even strategically including 「媽咪: mommy」 or even 「阿母 a-bú: mother （台）」 to engage the student. (Real educators do this very often, and students tend to really like it when they get “fun fact”-style local flavour!) I have not seen anyone attempt to introduce any of this design into AI-generated learning materials, and I suspect this is why they always come across as being so bland and mushy. Instead, the AI-generated materials are creating only rote practice items (which is why their prompts typically include things like “limit the generated text to use only vocabulary as published in the prep materials for such-and-such language proficiency exam.”) This kind of practice is, indeed, useful, but it's debatable whether it's measurably more useful than just spaced-repetition with flashcards.

Now, contrast these materials with primary texts (i.e., written language artefacts produced for an audience of native speakers.) Primary texts are often very difficult to incorporate into language learning, especially for languages like Chinese. This is probably because at the introductory level, the materials simply aren't dense enough for an adult learner, and at the advanced level, probably because these materials are far too challenging given the amount of specialised terminology and vocabulary used. (There are, in fact, very appropriate materials that sit between these extremes, such as news magazines or short stories written for middle schoolers, but these materials can be hard to access.)

The benefit of the primary text is that it is very close to the actual goal of the learner: I really don't want to read a story about a lost dog, and I only do it, because with enough practice reading such drivel, I might eventually read ‘Dream of the Red Mansion’ or ‘Red Sorghum.’ As a consequence, what most learners will reach for are “graded readers” which are adaptations of well-known works with simplified language and grammar. I'm on the fence with how well AI can create these for us. On the one hand, there is a pedagogical and creative dimension to producing a good graded reader. The former may be possible to approximate with additional prompting (“use only vocabulary from this list; use only grammatical structures familiar to a learner at this tested level,”) but I'm not sure about the latter. The reader is probably losing a lot when we simplify Gandalf to ‘Run away now!’

So while I'm quite hopeful that AI technologies can improve language learning, this kind of tool just doesn't seem to add anything to what already exists and is already much better.

The approach is just too obvious. I think it's too focused on finding a way to adapt something we know that AI can do well (generate grammatically correct text) to something we want to be able to do more cheaply or effectively (teach language learners how to read) without really considering how to solve this problem.

Agreed.

I have first-hand experience across five distinct AMD 7840U and AMD 8840U devices that near-perfect, out-of-the-box Linux-support (with stock kernels and no dodgy kernel flags!) is possible. This includes support for S0ix suspend.

https://news.ycombinator.com/item?id=43083669

I don't doubt it when people recount their bad experiences with AMD devices; however, my experience should serve as an existence proof that it's not a universal experience.

In the case of each device mentioned in the comment above, I followed a standard installation procedure from an Arch installer USB. I use only stock kernels: linux, linux-lts, and linux-zen. For almost all of the devices, the only kernel flags I pass are for enabling hibernate or handling FDE. (In one or two cases, the devices have portrait displays that have been installed for use in landscape-orientation. These need an `fbcon=rotate:…` kernel flag.)

In all but one case (the OneXPlayer X1 Ryzen) everything (except fingerprint readers) works flawlessly. In the case of the OneXPlayer X1 Ryzen, there is an intermittent issue with hang on suspend, but that may have gone away with a recent kernel update. If not, I'll probably come back to this blog post and see what I can do…

It can be really hit-or-miss, and it can be really hard to debug errors like in the post.

A lot of workarounds that are suggested for various issues are also not really viable. Some of the workarounds involve turning off different power-saving modes; however, the point of enabling sleep is often to increase the amount of usable time between charges, and turning off these power-saving modes can often dramatically shorten battery life.

But getting sleep to work (even S0ix!) is not impossible.

I have a bunch of handheld AMD 7840U and AMD 8840U devices that I have installed Arch Linux on: GPD Win Max 2, GPD Win Mini, GPD Win 4, Minisforum V3, OneXPlayer X1 Ryzen. These devices were not designed with Linux support in mind. I would be very surprised if the companies that made them ever tested them with Linux. Yet with just a small amount of work (generally fiddling with `/proc/acpi/wakeup` and `/sys/devices/*/*/*/power/wakeup` to disable sources of spurious wakeups,) I have gotten essentially flawless S0ix support (… on all but the newest OneXPlayer X1 Ryzen.)

(In general, out-of-the-box stock Linux kernel support on these devices is fantastic. Touchscreens work, pen input works, wifi and Bluetooth work well. The only gap I've seen is fingerprint reader support.)

I suspect that given how small these manufacturers are (and how small their production batches must be,) there's much less extreme-customization and tight-integration of components. This is visibly evident in the form-factors of these devices, which many millimeters thicker than they might otherwise be. (Of course, these devices are primarily advertised to a gaming audience who are eager to avoid the thermal-throttling that happens with ultra-thin devices like Surface Pro…) I partially suspect that the lack of extreme-customization, the lack of tight-integration, and the smaller production batches means that the manufacturers make much more conservative choices in components. Maybe this explains the exceptional Linux support?

This is not a comment that correctly describes how these two entities realistically operate and interact. I don't know why people keep repeating this as though it were insightful.

Whatever your own position on this matter may be, it is important that we factually describe the positions of the parties directly involved.

Hopefully we can use this as an opportunity to spread a more accurate description of the dynamics at play.

For reference, here is a speech by William 賴清德 (https://en.wikipedia.org/wiki/Lai_Ching-te) outlining the position under which he operates. The phrasing he uses is one that is consistent across all of his public remarks; consistent with remarks by Louise 蕭美琴(https://en.wikipedia.org/wiki/Hsiao_Bi-khim) and other close associates of William 賴清德; consistent with remarks made by 蔡英文 (https://en.wikipedia.org/wiki/Tsai_Ing-wen); and consistent with how the structures in Taiwan have operated over at least the last decade.

《賴清德就職演說：兩岸「互不隸屬」》: https://youtu.be/oLO5bYF8lDs?t=139

The relevant remark is: 「由此可見，中華民國與中華人民共和國『戶不隸屬』」

Here is my (manual) translation: “From this it can be seen that the Republic of China and the People's Republic of China are ‘not subordinate to each other.’”

Here is a Whisper-generated transcript of the entire speech, with an OpenAI generated translation inline. I skimmed the translation, and it adequately conveys the speaker's meaning and intention. (It does fail to convey the delicacy, careful phrasing, and specific rhetorical choices made by the speaker that are extremely clearly visible in the Chinese. However, this aspects are harder to convey if you don't have minimal prior knowledge on this topic.) https://pastebin.com/fGxHUpXN

It is true that there are historical positions, (historical) on-paper claims, and even a variety of differing positions and lively debate on this issue across all of the populations involved. But the conclusion one might draw from the comment above is wholly incorrect. It simply isn't the framing within which William 賴清德 and associated parties are actually navigating this issue.

I think you may want to clear the environment (e.g., of `SSH_AUTH_SOCK`) as well as isolate in a PID namespace as well. I also reflexively `--as-pid-1 --die-with-parent`.

    bwrap --dev-bind / / --clearenv --tmpfs ~ --unshare-pid --as-pid-1 --die-with-parent ssh terminal.shop

The historical background about tabular displays of quantitative information is very interesting. I imagine it must have been fun think deeply about this problem.

Unfortunately, the API design in the example is just not very good:

    (
       GT(simple_table, rowname_col='Name')
      .tab_header(title='Names, Addresses, and Characteristics of Remote Correspondents')
      .tab_stubhead(label=md('*Name*'))
      ...
    )

Of course, for ad hoc, in-a-notebook, cut-and-paste/written-from-scratch use, the API design doesn't really matter that match. Usually, users will readily memorise the required incantations then fiddle with the result until they get what they want or they give up.

It's probably the case that for most tools that produce visual outputs, a majority of users are creating things in this style. (There are, e.g., millions of casual Matplotlib users out there.) But programmatic use is not too far off. Tools that produce visual outputs (even those as formally rigidly at display tables,) are often subject to consistency requirements, which directly implies programmatic use.

So, when I discover that my colleagues and I have six tables across three notebooks that need a consistent look, and I decide to interact with this tool programmatically, am I expected to write…?

    def standard_table(source, /, rowname_col, header_title, stubhead_label, weight_columns):
      return (
        GT(source, rowname_col=rowname_col)
        .tab_header(title=header_title)
        .tab_stubhead(label=md(f"*{stubhead_label}*"))
        .fmt_integer(columns=weight_columns, pattern="{x} lbs")
        ...
      )

    standard_table(simple_table, rowname_col='Name', header_title='Names, Addresses, and Characteristics of Remote Correspondents', stubhead_label='Name', weight_columns='Weight')

    def format_table(weight_columns):
      return (
        tbl
        .tab_stubhead(label=md(f"*{tbl.stubhead.label}*")) # what if not present?
        .fmt_integer(columns=weight_columns, pattern="{x} lbs")
        ...
      )

    format_table(
      GT(simple_table, rowname_col='Name')
      .tab_header(title='Names, Addresses, and Characteristics of Remote Correspondents')
      .tab_stubhead(label='Name')
      ...
    )

     class StandardTable(GT):
       def tab_stubhead(self, *a, **kw):
         # inspect.signature.bind(...) # ...
         return super().tab_stubhead(*a, **kw)

    StandardTable(...)

That makes sense.

My assumption prior to purchasing these devices was that AMD support would be the main reason for me returning them.

I was quite surprised to see just how well everything worked, and, in fact, these devices have given me much less trouble than my i7-1165G7 Dell XPS 13".

In fact, these devices has worked so well that it's really made me more excited about consumer technology and new laptops and portables than I have been in a long time!

I am worried that I may be mistakenly generalising this to all AMD devices. I have even been considering ditching Intel for my next laptop upgrade…

I am very surprised to hear reports of poor Linux support for AMD devices. I recently purchased a GPD Win Max 2 [1] and a GPD Win 4 [2] for work. Both devices are AMD 7840U with Radeon 780M (and 64 GiB RAM at 6400 MT/s since I saw what appeared to be instability at 7500 MT/s.)

I installed Archlinux on both and the support has been very solid with the `linux-lts` kernel and only one kernel flag (`amd_pstate=guided`.)

The following work: touchscreen; suspend (s2idle, which correctly enters S0ix state giving ≥100h battery life suspended but requiring I disable entries in `/proc/acpi/wakeup`); audio; media keys; game controls; wifi; Bluetooth; USB; SecureBoot (with `systemd-boot`); webcam (available only on Win Max 2.) The TPM2 works, but I haven't done anything with it yet (and it's an fTPM.)

Battery life is quite good (8~10h with the `acpi-cpufreq` `powersave` governor.) If I dial down the TDP, I can eke out even more battery life than that. If I dial it up to performance, I get 1~2h of battery life (and the exhaust from the fan port runs very hot.) I plan to try the `amd_pstate` governor (kernel flag `amd_pstate=active`) once support is available in the LTS kernel, though I like being able to control TDP via the CLI with AUR packages like `ryzenadj-git`. (This sounds like a lot of work, but I have a short shell script where I dialed in the right `cpupower` and `ryzenadj-git` commands optimal for my use.)

Hibernate works perfectly on the Win 4; I haven't tested it on the Win Max 2 (since s2idle works so well!)

On the Win 4, I can't get `xrandr` modes at anything other than native resolution (1920×1080) to work, but I am content with scaling via `xrandr --scale-from`. All resolutions display properly on the Win Max 2. Rotation works perfectly on both devices, but I rotate manually (and I haven't bothered to see if there is a gyroscope sensor.) There is minimal to no screen tearing. The displays present to the OS as landscape (i.e., rotation is not necessary from the Linux console.)

The only hardware that outright doesn't work are the fingerprint readers (but I believe Linux support for fingerprint readers is limited to very few devices.)

In short, my recent experience with AMD 7840U/Radeon 780M has been nothing short of amazing, and these devices have been an absolute pleasure to use.

[1] https://www.gpd.hk/gpdwinmax2

[2] https://gpd.hk/gpdwin4

[3] https://aur.archlinux.org/packages/ryzenadj-git

> A Python program is a sequence of statements

Here's a Python program that contains no statements: 42.

It is true that Python's grammar features a statement/expression dichotomy, unlike many other tools. If we want to speak to Python's grammar, we should make sure to consider two general eras—before and after the introduction of the PEG parser. The PEG parser was introduced to Python with PEP-617[3].

Let's consider first the grammar used in Python 3.8, prior to the PEG parser. You can find this in Grammar/Grammar[4]. As we can clearly see, there are a number of “entry points” for a well-formed Python programme[5]:

    single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
    file_input: (NEWLINE | stmt)* ENDMARKER
    eval_input: testlist NEWLINE* ENDMARKER

The Python parser considered a file input to be a “sequence of statements.”

But, of course, who cares?

Remember that the goal of an instructor is to present just the right level of detail that an attendee can do something useful. It is the case that the expression/statement dichotomy is useful, especially when considering the common expression⇋statement dualities we see in the grammar, but this is not a topic for day one, slide one of an intermediate/advanced course.

By the way, if we look at the PEG grammar, we see similar[6]. A `file[mod_ty]` input is comprised of `statements` (and an `eval[mod_ty]` input is comprised of `expressions`.)

Therefore, it is incorrect to say that “a Python program may contain statements or expressions.” Instead, we should say that the Python interpreter parses a simple, single-file Python program as a sequence of parser-level `statements` which my themselves be value-producing entities (which we might refer to as “expression”) or non-value-producing entities (which we might refer to as “statement.”) We might note that there are places where “statement” grammatical entities are invalid to use and places where “expression” grammatical entities are invalid to use, and that this “statement”/“expression” dichotomy is one that is found in other programming languages (but that there are other programming languages which are avoid this distinction.) We might further note that this dichotomy has affected evolution of Python be introducing dualities—for all but a few “statement” forms, there is an equivalent “expression” form. There may be contortions required to exactly match one to the other. (e.g., `while`) There are cases where, absent significant contortions, there is no dual (e.g., `try/except/finally/else` or `match/case`.) As a consequence, there is not considered to be a simple way to transform any multi-line Python programme into a single-line, single-expression equivalent. But, of course, probably nobody really cares. You've lost the class on slide one.

Next, it is true that semicolons can separate Python statements in some cases. It is important to note that, with the exception of silencing last-expression output in a Jupyter notebook, it is possible to never encounter the use of a semicolon in real Python code.

It is unfair to assume the author intends to convey that the execution of a Python programme is strictly in the order of the appearance of the lines of code in a file, without considering that function calls contain a body of statements which are executed only on function evaluation. Instead, we should interpret this to bullet point to mean that Python programme executed top→down with statements executed at runtime in a manner dissimilar to how C++ works. For example, `def f(): pass` is executable code in Python, and this statement is, in fact, executed. This is why we might argue that the “mutable default argument” problem is largely a matter of misunderstanding Python's execution model. We could consider that the “execution” of the `def` starting statement means the parsing and compilation of its contents into bytecode, rather than the execution of the contents directly. After all, even `f()` on `def f(): …` is not guaranteed to actually evaluate the body of `f` in all circumstances.

It is incorrect to suggest that the presence of features like `sys.meta_path` or `sys.path_hooks` invalidates the top→down nature of Python's execution model. Sure, you can implement an import hook that `exec(''.join(open(…).readlines()[::-1]), …)` but this could easily be considered a modification of the input rather than a modification of the `exec` mechanism. In fact, there are many ways to generate bytecode without passing through the `exec` machinery, but these are generally considered esoteric. If we consider only the `exec` machinery, then we will see that it will invoke the standard tokenisation and parsing process which will execute its payload line-for-line from beginning-to-end. Sure, we can say that the presence of import hooks mean that a programmer could subject a file input to preprocessing prior to passing it to the `PyRun_`/`PyEval_` mechanisms or that a programmer could generate executable bytecode in any arbitrary fashion bypassing these mechanisms… but it's going to be rare to encounter these… on the first slide… on the first day… of an intermediate/advanced Python course. So who cares?

This has already been pretty exhausting, but I think I have adequately demonstrated that it is not the case that “every statement [here] is a lie” and that we can casually dismiss the work of David Beazley as that of a “witch doctor” or “snake oil peddler.”

In fact, I would suggest that I have put forth some evidence that the original criticism belies a weaker and less thorough understanding of not only the Python interpreter but of the instructional process than its confident tone might suggest. I hesitate to suggest further.

It's definitely unfair to cast aspersions on the CPython core development team. Python is an established, mature language with a growing, increasingly diverse development team. It is bound to be the case that there will be new contributors who do not have as strong an understanding of the entire language and the entire interpreter. It is not the case that the CPython core development team is as untalented as you suggest. In fact, I would assert that there are programmers among the CPython core team who are some of the most talented people I have ever met. Many are experts not only in Python, but in C, in C++, and across all languages that they use in their work.

[1] https://github.com/dabeaz-course/python-mastery/blob/main/Py...

[2] https://github.com/dabeaz-course/practical-python/blob/maste...

[3] https://peps.python.org/pep-0617/

[4] https://github.com/python/cpython/blob/3.8/Grammar/Grammar

[5] https://github.com/python/cpython/blob/1663f8ba8405fefc82f74...

[6] https://github.com/python/cpython/blob/main/Grammar/python.g...

For context, the bullet points you reference are on slide 1-10 of the supplementary slide deck[1] and are provided as part of an accelerated review. This is a recapitulation of materials covered in the introductory “Practical Python” course in unit 01-02[2]. The “Practical Python” course is designed to be taken by attendees who have minimal experience with Python, including those who have minimal prior experience with programming at all.

In the context of an intermediate/advanced course, these are clearly being provided as an overall framing, and are not intended to be read as a precise description of Python's execution model They are, instead, intended to be glossed over (perhaps by an attendee who has somehow skipped the introductory course.) As a result, it would not be appropriate for these bullet points to discuss the finer points of Python's expression/statement dichotomy. It is clear that their intention is to express, with simplification, the general nature of Python's execution model and to distinguish it from tools which the attendee may already be familiar with (e.g., C++.)

There are a number of mistakes made in your own explanation, some of which have been highlighted by other posters. I will provide my own corrections to illustrate ① how distracting, pointless, exhausting, and useless a precise accounting of the mechanisms would be (especially in the context of this course and given the likely profile of a course participant) and ② that there may be some unearned confidence leading back to the source of this criticism.

I have my own criticisms of this course, which I have shared in another comment. But, as a personal aside, I have often found that, when I pit myself against the world—everyone else is stupid and wrong—it has provided me with a good opportunity for self-reflection. I have found a lot of personal growth in interrogating and questioning my own confidence and striving to find meaning and truth in my instructional work.

I agree that the average quality of Python instructional material is quite low. The language is very popular, and it's often pitched as a beginner's language or a language for those who are not (or do not want to be) professional programmers. Free (uncurated) platforms like YouTube and LinkedIn make it very easy to distribute poor quality material (and provide very weak feedback to encourage quality improvement.)

I strongly reject any assertion that David Beazley's materials, his instructional abilities, or his capabilities as a programmer are lacking. Having worked with David, I can provide testimonial to his skills (though his body of work speaks for itself.)

The example you highlight amounts to little more than nitpicking, and it suggests a fundamental misunderstanding of the instructional process. Trust me: I have taught this exact course (as well as “Practical Python”) to numerous corporate audiences.

I taught this course to corporate clients for three or four years before developing my own materials.

The course materials for this course and the introductory course (“Practical Python”[1]) are quite thorough, but I've always found the portfolio analysis example very hokey.

There's enormous, accessible depth to these kinds of P&L reporting examples, but the course evolves this example in a much less interesting direction. Additionally, while the conceptual and theoretical materials is solid, the analytical and technical approach that the portfolio example takes quickly diverges from how we would actually solve a problem like this. (These days, attendees are very likely to have already been exposed to tools like pandas!) This requires additional instructor guidance to bridge the gap, to reconcile the pure Python and “PyData” approaches. (Of course, no other Python materials or Python instruction properly address and reconcile these two universes, and most Python materials that cover the “PyData” universe—especially those about pandas—are rife with foundational conceptual errors.)

Overall, David is an exceptional instructor, and his explanations and his written materials are top notch. He is one of the most thoughtful, most intelligent, and most engaging instructors I have ever worked with.

I understand from David that he rarely teaches this course or Practical Python to corporate audience, instead preferring to teach courses direct to the public. (In fact, I took over a few of his active corporate clients when he transitioned away from this work, which is what led me to drafting my own curricula.) I'm not sure if he still teaches this course at all anymore.

However, I would strongly encourage folks to look into his new courses, which cover a much broader set of topics (and are not Python-specific)! [2]

Also, if you do happen to be a Python programmer, be sure to check out his most recent book,“Python Distilled”[3]!

[1] https://dabeaz-course.github.io/practical-python/

[2] https://www.dabeaz.com/courses.html

[3] https://www.amazon.com/Python-Essential-Reference-Developers...

Similar (but a far more limited implementation):

“Write a Git Client from Scratch in 25 Minutes”

https://youtu.be/xvzo_nV9PjU

The fonts at the bottom of page one and the top of page two (王漢宗中明體注音 wp010-05.ttf and 王漢宗中楷體注音 wp010-08.ttf) include pronunciation guidance using zhùyīn fúhào 注音符號 (also called “Bopomofo” 「ㄈㄅㄇㄈ」 after the first four consonants): https://en.wikipedia.org/wiki/Bopomofo

This is the phonetic system that is used most commonly in Taiwan.

Typically, phonetic pronunciation guidance is used only in educational materials. For native speakers, this means materials only for very small children. However, in Taiwan, it's not uncommon to see 注音符號 guidance to indicate when a word should be said with a non-Mandarin pronunciation. You'll see this, for example, in shops whose names contain a pun when using the Taiwanese Hokkien pronunciation.

There are other fonts that include pronunciation guidance in hànyǔ pīnyīn 漢語拼音, the phonetic system used most commonly in China: e.g., http://fonts.mobanwang.com/200909/5832.html

I don't think there are any fonts with pronunciation guidance in any of the other phonetic systems (e.g., 通用拼音, Wade-Giles, 國語羅馬字) but these have almost all fallen into disuse and appear only in old signage, historical place names, or in people's names.

(Presumably, if you are born in Taiwan, you get to pick how you want your named romanized… especially since you may want the spelling of your name to match that of your relatives. But are you allowed to pick any transliteration you desire?)

I am not a Cantonese speaker; however, in Mandarin, fonts with phonetic guidance are very common.

e.g., Hann-Tzong Wang's (王漢宗) free font collection[1] includes two typefaces with phonetic pronunciation guidance. These are wp{0..3}10-05.ttf and wp{0..3}10-08.ttf [2] As you can see from the filenames, there are actually four different font files for each of these two typefaces. The font files numbered {1..3} are for 「破音字」, characters with alternate pronunciation.

When a user types a word like 「給予」 (ㄐ一ˇㄩˇ/jǐ yǔ) for which there is an alternate, less-common pronunciation (ㄐ一ˇ/jǐ instead of ㄍㄟˇ/gěi for 給) they simply change the font for just the affected character to the variant with the correct pronunciation.

In the case of this Cantonese Font, the authors distribute a single .ttf (alongside a “phrasebook” .ttf whose purpose is not clear to me) and indicate in the Roadmap section of the website that ligature support must be enabled. If alternate pronunciations are common in Cantonese, then I suspect that they must use some ligature-based method. I would have to imagine there must be cases where this could be ambiguous, but I don't know how you would resolve those.

(In practice, just swapping the font on a single character works fairly well.)

[1] https://code.google.com/archive/p/wangfonts/

[2] https://dywang.csie.cyut.edu.tw/dywang/download/pdf/sample-o...

I recently wrote an autorunner[1] (like Entr[2] and Watchexec[3]) so I have some recent exposure to this problem. (I will be releasing it on Github shortly.) My autorunner allows running interactive programmes, so it is very sensitive to lingering child processes.

For the purposes of the autorunner, I use approach 1.1.3 (“always write down the pid of every process you start, or otherwise coordinate between A and B”) and leave it to the user to figure out what happens if the child process misbehaves with relation to any processes it starts.

However, I want to point out that approach 1.1.4 (“A should run B inside a container”) is easier to do than one might expect, and I'd like to plug one of my favourite utilities—Bubblewrap[4]. The Bubblewrap documentation says “[y]ou are unlikely to use it directly from the commandline, although that is possible” but I have built some amazing little tools from it.

Try the following invocation:

    bwrap --ro-bind / / --proc /proc --unshare-pid ps

As you can see, there's a lot you can do with this tool with significantly lower overhead than using Docker. It has been my goal for some time to extract some of the functionality of Bubblewrap into a Zsh extension to allow accessing these mechanisms with even lower overhead. I think the creation of namespaces is a missing primitive in Linux shells, and being able to quickly construct namespaced environments allows for a style of safe, robust, simple shell scripting. e.g., if you create a mount namespace to run your script, you can actually be looser about parameterising file locations (since the namespace can ensure everything is exactly where you want it to be.)

[1] https://fosstodon.org/@dontusethsicode/110019380909461936

[2] http://eradman.com/entrproject/

[3] https://watchexec.github.io/

[4] https://github.com/containers/bubblewrap

A sibling comment shared a short list of common characters with multiple pronunciations. If we include variants where the difference is only tone, then the list of common-use characters with multiple pronunciations gets pretty long.

This list is also changes on a regular basis. The following news article details updates to pronunciation guidance promulgated by the Taiwan Ministry of Education at the end of 2022: https://www.ftvnews.com.tw/news/detail/2022326L08M1

It is my understanding that these new guidelines result in mandated changes to educational curricula as well as standardised testing; they are not wholly advisory. It is also my understanding that these changes can be mildly controversial, because even when the changes are made for the sake of simplification, they lead to discrepancies between the educational standard and common spoken language.

Congratulations on your release!

Don't let my skepticism (below) to detract from your accomplishment.

I am deeply skeptical not only about this tool in particular but about these tools (e.g., Dash, Panel, Streamlit &c.) in general. I've seen these tools work well for very simple use-cases, but they seem to require some combination of…

· cooperating requirements (matching exactly what the demos do, so you stay completely within the “regime” these tools support)

· upfront knowledge of all requirements (so that you never fall out of this regime)

· the ability to reject requirements (so that you can keep yourself within the regime, though this means the tool is leading the business, not the business leading the tool)

· the ability to accurately identify the boundaries of the regime from the tool documentation and design

For very simple examples, it's may be possible to stay within these constraints, but even when you manage to successfully use these tools, you do so at high (and, in my view, often unacceptable) risk.

In general, the risk/reward for a tool like this is about being able to learn the tool almost immediately (which must also account for time spent trawling through the documentation or the Github issues or even the `dir(module)` to figure out how to spell that one thing you need.) Basically, if you can create something in 1 day that can last for 180 days before a requirement comes along that forces you to start over from scratch, you may be okay.

You certainly never want to become good at these tools; the more you invest in the tooling, the harder it will be to abandon it when it is no longer appropriate, and the more willing you will be to accept contortions that you will inevitably regret. You certainly never want to spend more than a trivial amount of time building something with these tools, since the risk of needing to rebuild or abandon the technology is simply too high.

(Personally, I simply don't think React is that hard to learn, that hard to teach, or that onerous to use…)

In particular, this tool's API seems particularly risky. e.g., in the example code, there is a section like this:

    cond(
        State.mode,
        element,
        cond(
            State.flag,
            other_element,
        ),
    )

I suspect, however, that this API is still being refined. After all, what if I wanted to trigger on two conditions? There are probably some sophisticated `metaclass` mechanisms wrapping the `State` attributes in some symbolic modelling, but SymPy shows us that this is not trivial to get right, and the Python object model constraints you in unpredictable ways. Without looking at the code, I suspect you cannot `cond(State.mode1 | State.mode2, element)` and probably have to create some sort of `State.mode1_or_mode2` subordinate data, which spirals out of control pretty quickly. And what if you want to do `cond(State.mode | mutable_global_mode)`? And so forth and so on.

There are so many structures and modalities for control flow and data flow (trivially expressible in React) that will become clumsier and clumsier to express in with this modelling, and unless you can anticipate that they won't arise, reject the business requirements that mandate them, or (with no regret) throw everything away and start over from scratch, you're going to have a good experience.

Bubblewrap is a surprisingly useful tool for general system administration tasks.

Even though the documentation claims that "[y]ou are unlikely to use it directly from the commandline, although that is possible," I use it as a helper tool in this mode very frequently.

This can be very useful for debugging since, for example, you can `bwrap --ro-bind / / --tmpfs ~ $SHELL` to get a "clean" shell in which you can isolate yourself from the effect of configuration dotfiles and can even `--ro-bind my-hosts /etc/hosts` to simulate certain system-level state (without requiring a full VM, heavier container, or root access.)

Of course, I've also written some simple shell scripts around `bwrap` to make this all a bit simpler (since this quickly reaches `qemu`-levels of argv proliferation.)

The admin fees you refer to are typically assessed in the context of large, institutional grants. These are grants from private organisations like the Gordon and Betty Moore Foundation, the Alfred P. Sloan Foundation, the Chan Zuckerberg Initiative, or government agencies like the US National Science Foundation. These grants have significant financial accounting requirements. There are also many other legal or operational costs associated with these grants.

When projects run these grants through parent institutions like universities, the typical admin fee is >40%. In some cases, it can be as high as 60%. Many projects are eager to enter fiscal sponsorship agreements with organisations like NumFOCUS, because so much more of their grant money goes to funding the work.

In the case of NumFOCUS, admin fees do not adequately cover the staff requirements to manage these grants. NumFOCUS "loses money" when servicing the administrative needs of these grants & it takes this responsibility onto itself solely for the betterment of the projects.

Rather than assess administrative fees similar to universities, NumFOCUS uses its other fundraising—corporate donations, event (i.e., PyData) sponsorship, individual giving—to finance its operations.

source:

- I serve on the NumFOCUS board of directors as its co-chair.

- I presented on this topic last year at the NumFOCUS Annual Summit to an audience of core developers from projects like Julia, Jupyter, Pandas, NumPy, AstroPy, &c.

- NumFOCUS budgets are public, and all of the above information can be corroborated from materials published on https://numfocus.org/

In the future, will it be considered offensive to suggest that someone’s posts look like they were generated by an ML model…?

I think audit hooks were a great idea and are part of a defence-in-depth strategy.

In fact, I even implemented the variant of audit hooks that informed PEP-551 and deployed those to production for a major platform.

I think that if you were to use audit hooks, you might want to combine them with a code signing mechanism. I worked on two of these for different platforms, and they add another strong layer of defence.

In light of that, the bypass in the article seems somewhat contrived, especially considering that there are at least three alternate techniques you could employ to accomplish the same goal that would be meaningfully more innocuous than using `ctypes.windll`. If you're going to use `ctypes` or `pywin32` or the like, then you might as well write a C-extension module to patch out the audit hooks directly (as Batuhan Taşkaya shows using a simple trampoline toy library I wrote, `libhook`: https://speakerdeck.com/isidentical/hack-the-cpython?slide=3...).

Better techniques:

1. Joe Jevnik and my `tuple_setitem` which uses poisoned bytecode and the lack of bounds checks on `LOAD_FAST`/`GETLOCAL`. (Pure Python.)

https://gist.github.com/llllllllll/b1ac68a6b77535a64c0b13bfd...

2. My `tuple_setitem` using `numpy` raw memory access via `numpy.lib.stride_tricks.as_strided`. (Requires only `numpy`.)

https://twitter.com/dontusethiscode/status/12513186248470855...

3. Using the `/proc` filesystem on Linux, which gives arbitrary intraprocess read/write access (independent of page permissions.) (Requires only `open(..., 'w')`.)

https://twitter.com/dontusethiscode/status/12818444226285854... https://gist.github.com/dutc/2cc5de0d2f8877b8f463b86e8bd5231...

There are also a couple of techniques you could employ to carry one of these payloads past code signing, some of which are very well known, like the insecurity of `pickle` deserialisation, and some of which are… less well known.

(I have also prototyped using the above exploits to "lift" C code into a Python interpreter, in case there are OS-level defences around `dlopen`.)

However, even taking these into account, I'm a big believer of the value of Python 3.8 audit hooks at PEP-551, but they are technique that requires quite a bit of extra work to effectively employ.

If you're interested in trying to implement audit hooks and these other mechanisms for locking down your execution environment (e.g., you want to mitigate exploits in Python systems, which may run as PID-1 in a containers, where these exploits may try to bring in malware that could exfiltrate data…) please feel free to reach out to me by e-mail or Twitter.

I would be happy to share more with any organisations that are large enough to consider locking things down at this level.