knz42

What do you think of the minimal phone? https://minimalcompany.com/

A guide for newcomers to software engineering, and for anyone curious about how LLMs are changing it. Built from interviews with senior professionals from the tech industry.

MLS (Message Layer Security) is the IETF standard (RFC 9420) derived from the Signal and Whatsapp end-to-end encryption protocols, with improvements.

Whoosh

OP was parodying JD Vance 's comments from last week.

This is a known bug in st when the default emoji font contains color emojis. Explained here: https://github.com/LukeSmithxyz/voidrice/issues/284

One of the simplest workarounds is to ensure that the default emoji font is black&white; the linked issue above suggests other workarounds.

Also in the official st FAQ: https://git.suckless.org/st/file/FAQ.html

(search for "when trying to render emoji")

Have you thought about simple iframes?

For example: https://random.vodka-pomme.net/

Is there a way to simulate capital tax? Percentage of taxable investments/cash?

A lot of the complexity comes from the lack of expressivity in languages to relate variables (or data structure fields) semantically to each other. If there was a way to tell the compiler "these variables are always accessed in tandem", the compiler could be smart about ordering and memory fences.

The idea to extend programming languages and type systems in that direction is not new: folk who've been using distributed computing for computations have to think about this already, and could teach a few things to folk who use shared memory multi-processors.

Here's an idea for ISA primitives that could help a language group variables together: bind/propagate operators on (combinations of) address ranges. https://pure.uva.nl/ws/files/1813114/109501_19.pdf

This is a hit piece. None of the angle taken there is specific to Telegram, it could be applied equally to e-mail.

The article says that BI doesn't affect employment % as if it is a bad thing.

The fact that it did not decrease employment either is a wonderful thing, and should be celebrated.

It means that Finland can use BI as a replacement for a number of other social services, and drive the cost of state support for unemployment significantly. This is excellent!

This is the approach proposed in http://doi.org/10.1109/TPDS.2015.2492542

(preprint: https://science.raphael.poss.name/pub/poss.15.tpds.pdf )

Have you tried hosted CockroachDB, aka CockroachCloud?

The title of the article ("My Go executable files are still getting larger") appears to be literally untrue, at least read as a critique of Go itself. If they are getting larger, it's because new code is being added, not because the Go runtime or compiler is degrading in some way over time.

Yes this is a fair assessment, although I find it surprising (and enlightening) that you refer to “a critique of Go”. At no moment was the intent to critique Go specifically; the entire analysis is made of observation of the results of combining Go with specific (and varying) amounts of source code.

In any case, based on this discussion I have decided to amend the title and emphasize in the conclusion that the absolute size of the code+data for a fixed amount of source code has decreased between go 1.15 and 1.16.

edit: This is relevant to this discussion: https://sneak.berlin/20191201/american-communication/

The use case where said binary is shipped to GCE instances hundreds/thousands times per day, for stress testing and unit testing of cockroachdb.

You said that " the article made the claim that 70% of space is wasted dark bytes" was "incorrect" with no further details

I wrote this because there was no mention of "waste" anywhere in OP.

a claim which is still in the article linked here. I am now supposed to argue against some revised article published elsewhere?

The article linked in this thread is a syndicated copy of an original article published elsewhere, as clearly stated by the attribution section at the bottom. It's reasonable to expect that changes to the original will only be updated in the copy with a delay.

all that matters is the total size of binary! How bytes are organized internally is irrelevant to this metric.

Not quite so if the task is to work on reducing the metric.

When the size is attributed to data/code that's linked to the source code, then we know how to reduce the final file size (by removing data/code from the source code, or reducing them).

When the size is non-attributed and/or non-explained (“dark”) we are lacking a control to make the size smaller over time.

For being somewhat familiar with the CockroachDB project, I doubt that that claimed performance difference is linked to the programming language. It's more something about mandatory 3-way (or more) replication upon every write, and several additional layers of protection against hardware failures, network problems etc which postgres do not have.

That's not what it says [...]

That claim was retracted a while ago already on the original version; the syndicated copy on the crl web site will be updated at some point.

Thanks Russ for that additional insight.

I would hope that anyone sensitive to networking costs at this level would be shipping around stripped binaries, so the growth in accurate DWARF info should not be relevant to this post at all.

Good point. I removed that part from the conclusion.

If you subtract out the strippable overheads and you take the "Dark + pclntab" as an accurate representation of Go-specific overhead [...] then the situation has actually improved markedly since Go 1.12 [...] Whereas in Go 1.12 the measured "actual program" was only about 40% of the stripped binary, in Go 1.16 that fraction has risen to closer to 55%.

Ok, that is fair. I will attempt to produce a new version of these tables with this clarification.

the stripped CockroachDB binary in question has gotten smaller since April 2019, and less of the binary is occupied by what the post calls "non-useful" or "Go-internal" bytes.

There's an explanation for that, which is that the crdb code was also reduced in that time frame.

tried to blame it partly on the Go compiler producing more bloated code over time

Where? The argument is _precisely_ that the growth is occurring in non-code areas.

partly on a mystical "dark area" which you don't understand

The _observation_ is that the growth is happening in an area of the file that's not accounted for in the symtable. That's what makes it "dark". It's not mythical: it's _there_ and you can observe it just as well as anyone else.

you mentioned superlinear growth only in the comment section

it's in the reported measurements.

and you didn't actually gather data or do experiments to prove or disprove any of the things you're claiming as a cause

The analysis is stating observations and reporting that the size is increasingly due to non-accounted data. That observation is substantiated by measurements. There's no claim of cause in the text!

Dark silicon is not used all the time - that's the key point.

In the same way, the functab data in Go is not used all the time either, only when generating stack traces.

Also since that original article from 2011 was published, the phrase "dark silicon" was extended to designate silicon IP which is not directly necessary for a given target application but is embedded for the occasional use by a niche in the same market.

In the same way, all languages (not just Go) embed constructs in generated code that are there only to handle edge cases in certain applications/deployments, and are inactive for 90%+ of executions.

The article made the claim that 70% of space is wasted "dark bytes"

This is incorrect. The claim is that the bytes are either non-accounted, or motivated by technical choices specific to Go.

What you're doing here is pretty much the same trick quacks [...]

Look the article has some measurements with numbers which you can readily reproduce on your own computer, and the methodology is even described. The main claim is that "it's unclear what these bytes are about". The previous claim that they were "non-useful" was retracted. The data is there, and there's a question: "What is this data about?"

The text is even doubling down by spelling out "there's no satisfying explanation yet".

outright disbelieving people like this is rather rude

We're not in the business of "believing" or "disbelieving" here I think? There's data, there's measurements, and there are explanations.

After my comments and that of others, Russ provided a more elaborate, more detailed (and at last, falsifiable in the positive, epistemological sense of the word) explanation deeper in the thread. Now we can make the work of looking into it and check the explanation.

your comment has the strong implication that Russ is outright lying

Your understanding is flawed then? There was no assumption of lies implied.

Go executable sizes haven't increased much in general.

Russ's example was just the "gofmt" program.

Perhaps the reason you're seeing increases in Cockroach DB is because you keep writing more code for Cockroach DB?

If that was the only reason, then the % overhead would remain constant-ish. But it is increasing. So there is a non-linear factor for _some_ go programs (like cockroachdb) and it's still unclear what that factor is.

What in my reply suggests I assumed nefarious intent?

Brad is a good person, it's the code we're talking about here.

The expansion of pclntab in Go 1.2 dramatically improved startup time and reduced memory footprint [...]

yes this is acknowledged in the OP

We (the Go team) did not “recompress” pclntab in Go 1.15.

There's now less redundancy in the funcdata, so in my book less redundancy = more compression.

We did not remove pclntab in Go 1.16.

Correct; it is not "removed"; instead the advertised size of the symbol has been reduced to zero. Maybe the data is still there, but it's not accounted any more.

Changed in the text. (The correction is already present in the original version of the analysis, and the cockroach labs copy should be updated soon)

we never claimed “pclntab has been reduced to zero”, which is presented in the article as if a direct quote.

Correct, there was indeed no such claim. Removed the quotes and rephrased that paragraph.

If the 73% of the binary diagnosed as “not useful” were really not useful, a reasonable demonstration would be to delete it from the binary and see the binary still run. It clearly would not.

1) the phrase "non-useful" was a mistake. There is no definite proof it is non-useful, as you pointed out. Corrected in the text.

2) see discussion below - the demonstration is more complicated than that, as removing 100 bytes where just 1 byte is necessary will break the executable in the same way as removing 100 necessary bytes.

I think the proper next step here is to acknowledge that the problem is not "usefulness" but rather accountability.

The big table seems to claim that a 40 MB Go 1.8 binary has grown to a 289 MB Go 1.16 binary. That’s certainly not the case. More is changing from line to line in that table than the Go version.

Correct. Added a note to emphasize this fact.

Overall, the claim of “dark bytes” or “non-useful bytes” strikes me as similar to the claims of “junk DNA”. They’re not dark or non-useful.

Let's forget about "non-useful", this was a mistake and will be corrected. The word 'dark' is still relevant however. The adequate comparison is not "junk DNA", but instead "dark silicon":

https://ieeexplore.ieee.org/abstract/document/6307773

We're talking about a general % of executable size that's necessary for a smaller % of use cases in program function.

I'm all for trade-offs, but IMHO they should be transparent.

This is effectively what is happening btw; the crdb binary went from 80MB to 200MB in the same time it took to make it twice as fast. The % growth in size is not a problem on its own; it's more the % size attributed to the program vs. the % size attributed to unclear purposes, that's a problem.

This code, if I'm reading it right, uses the symbol table and ELF section headers.

As explained in OP, the sum of sizes advertised in the symtable and ELF section headers does not add up to the final binary size. The shotizam tool is thus blind to that difference.

But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about?

if you do know, then pray, what is the answer to this question?

Strip removes the symbol tables and DWARF information.

But still, the sum of all the bytes advertised in symbol tables for the non-DWARF data does not sum up to the stripped size. What's the remainder about?

I am reminded of how early versions of MSWord were embedding pages of heap space in save files that were not relevant to the document being saved, just because it made the saving algorithm simpler. For all we know, the go linker could be embedding random data.