A Little Story About the `yes` Unix Command

https://matthias-endler.de/2017/yes/

by mre • 9 years ago

452 172 9 years ago

kccqzy3• 9 years ago

Really? You didn’t even mention reducing system calls? That’s basically what full_write does: try to output a whole buffer with one system call.

In your regular program, even with just a normal setvbuf call to set up block buffering would make a huge difference.

dmitrygr4• 9 years ago

That last rust example is less readable than modern template-metaprogramming variants of C++.

There is something elegant about being able to beat it out on speed with about 30 lines of C

a_t482• 9 years ago

Simply putting a comment above each function would make it bearable.

dmitrygr4• 9 years ago

not as readable as this faster variant

  #include <sys/uio.h>

  //2048 "y\n" s
  static char __attribute__((aligned(4096))) str[] = "y\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\ny\n";
  static struct iovec v[] = {
  	{str, 4096},			//adjust number of these to taste
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  	{str, 4096},
  };
  
  
  int main(int argc, char** argv) {
  
  	while(1) {
  		//scatter-gather write (on linux there is basically no way for write to write an odd number of characters, except EOF, so we need not check for partial writes
  		(void)writev(1, v, sizeof(v) / sizeof(*v));
  	}
  	return 0;
  }

flukus3• 9 years ago

And the naive "readable" version is half the speed of python. Makes you wonder how typical that would be in real code?

glandium9• 9 years ago

Fun fact:

  $ python yes.py | pv -r > /dev/null
  [11.2MiB/s]

  $ python3 yes.py | pv -r > /dev/null
  [4.95MiB/s]

As for why the naive rust version is slower, it's because without adding a BufWriter in rust, stdout is line-buffered, so each line emits a write system call, while with python, stdout is buffered. Python 2 emits writes of 4096 bytes, and python 3... 8193 bytes (edit: not a typo, this is 8KiB + 1). That's the likely cause for it being slower.

Edit: A minimal version of the naive rust version would be:

  fn main() {
    loop {
      println!("y");
    }
  }

On the same machine as with the python tests above, I get:

  $ ./yes | pv -r > /dev/null
  [4.81MiB/s]

which is actually as slow as python 3, despite doing 4 thousand times more system calls.

A version with buffering would look like:

  use std::io::{stdout,BufWriter,Write};

  fn main() {
    let stdout = stdout();
    let mut out = BufWriter::new(stdout.lock());
    loop {
      writeln!(out, "y").unwrap();
    }
  }

And produces 129MiB/s on the same machine. And that's strictly doing the same as what the python version does (with a default buffer size of 8KiB, apparently).

And fwiw, on the same machine, both GNU yes and the full rust solution from OP do 10.5GiB/s.

kibwen7• 9 years ago

For a program like this, execution is I/O bound rather than CPU bound, so even if Python is generally less CPU efficient than Rust the effect is overwhelmed by a different I/O strategy. It's just like how Node has greater max throughput for trivial server workloads than a naive C implementation.

agwa2• 9 years ago

The main function is egregiously overcomplicated.

This is how I'd write it instead:

  fn main() {
        match env::args_os().nth(1) {
                None => write(b"y\n"),
                Some(arg) => {
                        let mut arg = to_bytes(arg);
                        arg.push(b'\n');
                        write(&arg);
                }
        }
        process::exit(1);
  }

It's just as efficient and far more readable. It avoids Cow, a lambda, and the chain of combinators.

sinistersnare• 9 years ago

Doesn't this only print once, and not over and over, like `yes` should?

kibwen7• 9 years ago

There is something elegant about being able to beat it out on speed with about 30 lines of C

What is this referring to? If it's referring to GNU C, the linked C implementation is about 100 non-comment lines, and is three times slower on the author's machine than the Rust version.

dmitrygr4• 9 years ago

see sister comment

mrighele2• 9 years ago

As other have pointed out elsewhere what matters here is not much the performance of the language but how much you can avoid calling IO code by making the path as straightforward as possible. You can get comparable performance ( 6.5 GB/s vs 8. GB/s on my machine) with 10 lines of python code, which probably took me the same time to write as the original rust version.

e12e• 9 years ago

I'm sure a lot of people would like to see the code for comparison. I know I would.

billsmithaustin• 9 years ago

Those performance improvements make the code more complicated too. How fast does the yes command actually need to be?

bluedino• 9 years ago

This could be applied to something like memset, another deceiving simply function

kccqzy3• 9 years ago

The git commit that sped things up mentioned using yes to generate test data.

loeg2• 9 years ago

Yeah — probably it doesn't matter how fast the yes command is. It will be faster than the program prompting for y/n.

FreeBSD's was nice and simple until someone decided to make it faster in r319897.

vacri2• 9 years ago

The yes command will repeat whatever string you give it, so it's useful for load/traffic testing. Speed helps this particular use-case.

musage• 9 years ago

In this case it's probably just for fun and exercise, but generally don't think of speedy code as extending more effort, but as making the computer waste less effort. Well, unless the program isn't blocking anything any just waiting for the network, and other such cases. But any code that actually does run, especially in a loop, doesn't run in a vacuum, so I'd rather err on the case of speed. These tools don't get rewritten every 2 years in a new framework, which probably helps. Why not hone them if they're going to be that heavily used?

jsolson• 9 years ago

Ah, so, this raises an interesting question: what do people use yes for?

One answer (a bad answer) is "generating artificial load".

This makes the I/O strategy oddly relevant -- different strategies result in different numbers of syscalls and (potentially) kernel lock acquisitions.

Couple this with someone trying to benchmark something's behavior with an antagonist load, and the story gets downright painful to contemplate in terms of confusing results.

Source: in my younger years, I did this. In more recent years, I've seen other engineers do it.

VintageCool2• 9 years ago

I would desire the yes command to be as slow as possible while still performing its basic function of automatically confirming questions that come up during an installation.

This is because I clusterssh into 40 machines and hey I haven't formally accepted them into my known_hosts file yet so I type "yes" to acknowledge them, but whoops I had already accepted two of them so now they are spitting out the letter 'y' as fast as they possibly can and now I have to wait for all of that output to transfer over the wire onto my machine despite pressing ctrl+c a minute ago.

Blackthorn• 9 years ago

Then just use yes | head -50? There's no need to artificially slow it down when there are more reasonable means of capping output than relying on SIGINT.

boomlinde3• 9 years ago

I'm not familiar with clusterssh, but assuming that it passes flags to ssh you can add `-o StrictHostKeyChecking=no`.

Of course, the parameter does what it says, but the security implications are really the same as for blindly yessing your way through. The best is to pre-populate your hosts file using ssh-keyscan.

mrep• 9 years ago

I know right. The only time I have ever used it has required just 1 y/n and a new line. In that case, these "performance improvements" might make it slower for real world use cases.

13of40• 9 years ago

I guess a stretch goal would be to make a "shouldi" command that can consume more y's per second than yes can produce. Of course at that point the shell itself would probably become the bottleneck.

glandium9• 9 years ago

The shell can't be a bottleneck. If you run `yes | shouldi`, all the shell does is setup the pipe, give one of its ends as stdin to shouldi, and the other end as stdout to yes.

kccqzy3• 9 years ago

The shell does not even enter into the picture. The shell sets up the pipe and starts your programs. The `yes` program writes its output to stdout which is buffered in kernel and then directly read by your hypothetical `shouldi` program.

yjftsjthsd-h• 9 years ago

yes >/dev/null ?

cannikin• 9 years ago

Funny, I just learned about this command a couple of days ago as a simple way to max out your CPU. I was trying to drain the battery on my Macbook Pro and running 4 of these at the same time did the trick nicely. Redirected to /dev/null and run in the background: "yes > /dev/null &"

blibble2• 9 years ago

all the reddit users from the linked article missed the SIGPIPE trick!

you don't need to check the return value from write() as your process will be terminated with SIGPIPE if it tries writing to a closed pipe.

saying that, none of them check the return code correctly: if the consumer only reads a byte at a time you could eventually get 'yyyyyyyyyyyyyyyyyy' (without any newlines)

quite impressive that so many implementations of "yes" have the same bug :)

dmitrygr4• 9 years ago

please demonstrate that consumer.

on linux there is no way to do this without your consumer being a kernel driver that actually [purposefully accepts one byte at a time. all file io and pipes have basically no way to accept one byte only (except EOF)

blibble2• 9 years ago

you wouldn't see it on Linux as pipes are implemented using complete pages (always powers of 2), but there's probably some OS out there with a different implementation where you can set the buffer size to be an odd number, and then you'd see the bug with plain simple cat

I didn't say you'd likely see it in practice :)

caf• 9 years ago

You can't rely on SIGPIPE to terminate it or your program will fail to terminate on error if its output is redirected to a disk file.

electrum• 9 years ago

write(2) is a blocking call (unless O_NONBLOCK is set), so you should never have a partial write while the descriptor is connected to something valid.

snikch3• 9 years ago

I use the yes command to defrost my lunch. I open up a couple of tabs running

yes > /dev/null

Then place my frozen lunch on the back of my macbook. Give it an hour or so and boom, defrosted.

userbinator• 9 years ago

Reminds me of http://thedailywtf.com/articles/Just-a-WarmUp , a trick which I also discovered independently many years ago and used to keep my fingers warm while attending lectures in a nearly-unheated room.

taneq3• 9 years ago

Prior art for https://xkcd.com/1172/ ?

Bromskloss4• 9 years ago

I open up a couple of tabs

So, you're basically thawing your lunch manually. Have you considered the "parallel" command?

snikch3• 9 years ago

Hrm, no I hadn't. I don't know much about this command. Could you give me a (thawing) example?

kbaker• 9 years ago

Geez man, at least mine some cryptocurrency.

Then your hot lunch will buy you another hot lunch down the road!

knowingathing• 9 years ago

that's interesting. i open and close the mac slack app 5 times to defrost my lunch.

Abishek_Muthian• 9 years ago

Lol I defrost the chocolate like this by placing it near the air vent. But care should be taken to make sure that the wrapper doesn’t touch the chocolate while peeling off, because I don’t think the hot air from the vent is hot enough to kill all harmful microbes.

snikch3• 9 years ago

I think this method is much more healthy for you.

TFortunato2• 9 years ago

Hah, nice! Somewhat related, I once worked on a project that used a high reliability PC meant for extended use in "extreme" outdoor environments. One of the issues they (manufacturer) worried about was the pcb and solder joints experiencing thermal fatigue fron lots of seasonal and night/day temperature cycles.

Their ingenious solution was to always run the system towards the warmer end of its spec, and so it included a program that would monitor the temperature inside the case, and would spawn/kill a bunch of threads doing compute intensive math in order to keep the temperature constant when the users workload wasn't enough!

TheSpiceIsLife• 9 years ago

That's amusing.

The laser cutter I operate simply has reverse cycle air conditioner mounted in one of the cabinet doors.

andrewedstrom• 9 years ago

If only mining cryptocurrency had been a thing back then...

Asooka• 9 years ago

Couldn't they slow down the cooling fans? Or was it a passively cooled PC?

LambdaComplex• 9 years ago

I really don't know whether or not your comment is serious, but I damn sure want it to be

raverbashing• 9 years ago

I just open a webgl page, seems more effective

crehn• 9 years ago

Just for fun:

GNU true.c: https://github.com/coreutils/coreutils/blob/master/src/true....

OpenBSD true.c: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/true/tr...

glandium9• 9 years ago

FWIW, GNU true.c is that way because the GNU policy is for programs to all have --help and --version arguments.

Other fun fact, in some UNIXes, true looked(looks?) like:

  #!/bin/sh
  # Some multiple line copyright notice from AT&T

Yup, that's a shell script with only comments, all of which are a copyright notice for what is, essentially, an empty file. I saw that on Solaris something like 15 years ago. I'm sure it made its way to other flavors of UNIX.

kibwen7• 9 years ago

Now you've got me wondering what it would mean to try and enforce the copyright on an empty file. You'd think there would be some prior art, right? :)

ckuehl• 9 years ago

There are some amusing notes about the /bin/true copyright claim here: http://trillian.mit.edu/~jc/humor/ATT_Copyright_true.html

noobermin• 9 years ago

That comment on the bsd one from theo is rather funny.

wahern4• 9 years ago

I'm surprised that neither GNU yes nor GNU cat uses splice(2). Here's what I get. Note that ./rust is the final Rust program from the article modified to print stats and exit after a set byte count[1], ./splice is a simple C program that uses splice to copy the input string from a generated anonymous temporary file to stdout, and ./consume uses splice to move data from stdin to /dev/null. In all tests the default string of "y\n" is used.

  $ ./rust | cat >/dev/null
  5.220 GB/s (8589934592 bytes in 1.645 seconds)

  $ ./splice | cat >/dev/null
  14.514 GB/s (8589934592 bytes in 0.551 seconds)

  $ ./rust | ./consume 
  9.312 GB/s (8589934592 bytes in 0.922 seconds)

  $ ./splice | ./consume 
  25.118 GB/s (8589934592 bytes in 0.318 seconds)

As you can see, cat(1) is the real bottleneck.

splice is Linux-only. Most systems have sendfile(2), including Linux, but I didn't test it. The implementation and semantics of sendfile vary across platforms.

[1] I did this before I wrote ./consume (which takes an optional byte count limit), having assumed that GNU cat was using splice. As cat doesn't have a way to limit the number of bytes read/written, and other tools like head or tail definitely don't use splice, the simplest way to limit the benchmark without introducing a bottleneck was to have the producers themselves print stats and exit. The relevant diff is

    + let mut count = 0u64 as usize;
    + let bytes = (1u64 << 33) as usize;
    ...
    -while locked.write_all(filled).is_ok() {}
    +while locked.write_all(filled).is_ok() {
    +  count += filled.len();
    +  if count >= bytes {
    +      break;
    +  }
    +}

Y_Y• 9 years ago

Forgive me if I'm missing something subtle, but aren't the first two examples Useless Use Of Cat?

jwilk3• 9 years ago

Show us the source of your programs. :)

pwg• 9 years ago

I'm surprised that neither GNU yes nor GNU cat uses splice(2). ... splice is Linux-only.

The fact that splice is Linux only is most likely why. GNU programs are portable to many different OSes, many of which no longer exist in any real form. They try to not use any 'single OS' specific feature if at all possible.

halfelf• 9 years ago

Once read similar things before, on reddit: https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

sgt• 9 years ago

On my Mac I'm getting

$ yes | pv -r > /dev/null

[34.9MiB/s]

aleden2• 9 years ago

hmm going in the direction of [binary] instead of [shell script], why not just go further and ditch the whole C stdlib by passing -ffreestanding and then compiling

  void _start(void) {
    exit(0);
  }

pixelbeat• 9 years ago

The GNU variant was discussed recently at: https://news.ycombinator.com/item?id=14542938

The commit that sped up GNU yes has a summary of the perf measurements: https://github.com/coreutils/coreutils/commit/3521722

yes can be used to generate arbitrary repeated data for testing or whatever, so it is useful to be fast

feelin_googley• 9 years ago

   while[1;`0:"yes\n"]

nodesocket• 9 years ago

    main(argc, argv)
    char **argv;
    {
      for (;;)
         printf("%s\n", argc>1? argv[1]: "y");
    }

Is beautiful, readable, and minimal. The "optimized" Rust version is complicated and over 50 lines of code. At what point does performance optimization go too far?

1_2__4• 9 years ago

As a non-Rust developer my first thought upon seeing the optimized version was “On the other hand let’s not visit Rust, ‘tis a silly place.”

LEGOlord208• 9 years ago

There was a lot of things about Rust that made think it was weird before I actually gave in and tried it. Now I adore Rust. You should try it.

azernik2• 9 years ago

Since it can be used to repeat arbitrary data, it is liable to be used in performance-sensitive tasks (mostly along the lines of pumping dummy data to a program or piece of hardware that's being benchmarked).

trungaczne• 9 years ago

This is an unfair comparison. For starter this version too is unoptimized, so you need to compare this with the first Rust code, which is about the same size.

Moreover a lot of the "bloat" that comes with Rust code isn't because it's more performant. Neither C++ and Rust try to be faster languages (after all, the optimization tricks you can do with your program is limited to what you can do with Assembly and its cousin, C), they try to be safer by providing more abstractions and restrictions (compile time checks).

Myrmornis• 9 years ago

The author says "no magic here" for the C version:

  for (;;)
    printf("%s\n", argc>1? argv[1]: "y");

but it's not totally obvious to me whether the argument to printf would be evaluated on every iteration of the for loop or not. Does the compiler know that those don't change, and is the answer to that question fairly basic C knowledge or not?

pkaye• 9 years ago

The compiler can assume that argc will not change within the loop so it can optimize it. I just looked up the assembly output from gcc and it pulls the argc>1 outside of the loop and replaced the printf with puts. So something like:

  if argc>1
      for (;;) puts(argv[1])
  else
      for (;;) puts("y")

The replacing of printf with puts is based on gcc having specific knowledge about the printf library function.

ben_bai• 9 years ago

This is exactly the OpenBSD implementation.

jlebrech• 9 years ago

maybe it's supposed to be slow, wouldn't a faster 'yes' spam stdin much faster? you only need to hit yes occasionally and faster than a second

sabujp• 9 years ago

tldr; watched pootie tang

dreta2• 9 years ago

Can anybody explain to me what's this syntax? This is the first time i see anything like it, and i've been programming in C since i was a teenager.

    main(argc, argv)
    char **argv;
    {
    }

ecma• 9 years ago

That's a K&R (Kernighan and Ritchie) style function declaration. Compilers still support it but the version you'd be more familiar with (ANSI C style) has been standard since at least the late 80s IIRC. ANSI C was standardised in 1989 but that process had been in progress for something like 5 years beforehand.

colejohnson66• 9 years ago

Does it have to do with how, in BCPL (one of C’s ancestors), everything was basically an integer? So everything is an integer unless told otherwise?

boomlinde3• 9 years ago

main and argc are declared as int (technically auto int) by default. argv is declared and given its type between the function head and the function block, which is perfectly legal but quite unusual these days

This syntax is probably nice if you use a line oriented editor like ed

dreta2• 9 years ago

I was mostly talking about the odd re-declaration of argv. Everything else is quite clear.

FRex• 9 years ago

It's not that hard or unlikely to run into it in old C code. The game Nethack is still being developed and it's programmed this way too (anything I looked at in its code at least).

Annatar3• 9 years ago

  env::args().nth(1).unwrap_or("y".into());

this ridiculously complicated syntax to perform such a simple thing is why I will never accept Rust. What a clumsy, ugly language. I’ll just stick with learning ANSI common LISP, that pays immediate dividends.

0T-T0• 9 years ago

The only difference from

  expletive = 'y' if sys.argc == 1 else sys.argv[2]

is the memory safety guards, which happens to be the key feature of Rust

Annatar3• 9 years ago

But what if one doesn’t care about memory safety because one has other languages at one’s disposal, like shell, AWK, Lisp? What benefit does Rust bring then? And at what price?

And who is to say that the programmers of Rust built a perfect language that always generates code which is bounds-safe? I question that as I know from experience that no human has ever written 100% correct code 100% of the time; even machines aren’t capable of achieving that. Therefore, I hold that the entire promise of Rust is flawed. I don’t like Rust one bit.

simias2• 9 years ago

It's fairly idiomatic so you get used to it but if you prefer something a little more C-ish you can write it like that (as a side note the code in TFA forgets to import std::env and BufWriter):

    use std::env::args;

    fn main() {
    
       let expletive =
            match args().nth(1) {
                Some(arg) => arg,
                None => "y".into(),
            };
    
        loop {
            println!("{}", expletive);
        }
    }

This is the equivalent to the unoptimized C version. I would argue that it's a little more readable too, substituting "loop" instead of the idiomatic "for(;;)" and the more verbose match syntax instead of the ternary (the terser rust equivalent being the `unwrap_or` you seemed to dislike).

Personally I think I'd use the "unwrap_or" version, when you're familiar with the language it's completely transparent, easier to parse and expresses intent better I think. For an outsider I can see why it would look like a strange incantation though (but the same could be said about ternaries or CL's "do" construct for instance).

The only thing I'd deem inelegant here is the `"y".into()`. It's probably not obvious what it does and why it's necessary to somebody not familiar with Rust.

Annatar3• 9 years ago

That example is indeed more readable, but it still reads like methods on objects which is a fundamental issue for me. Any non trivial program written in an object-oriented paradigm eventually becomes incomprehensible, whereas the functional approach doesn’t rely on the state machine model. Two different approaches. I didn’t care for objects back in 1990 and I certainly care even less for them now. I upvoted your example in appreciation for the effort you went through though.

dnel• 9 years ago

Interesting, but is there actually a practical benefit of having GigaBytes of the word yes being generated or is this all just optimisation porn?

kqr2• 9 years ago

I thought about that too, but I suspect it may be some sort of race to idle thing.

unkown-unknowns• 9 years ago

Someone else ITT pointed out a good reason:

yes can be used to generate arbitrary repeated data for testing or whatever, so it is useful to be fast

dcosson• 9 years ago

Seems like it might even be better if yes was rate limited to a couple of lines per second or less. It would be more than enough for its intended use and when users inevitably run it in a shell to see what it does it wouldn't generate tons of output.

It seems like a violation of "do one thing well" to use it for generating data for testing, isn't dd and /dev/zero a better way to do that?

jwilk3• 9 years ago

Curiously, yes(1) is not standardized by POSIX.

Are there any UNIX systems that don't have it?

xurukefi• 9 years ago

  import sys

  while True:
      sys.stdout.write("y\n" * 2**16)

gives me over 4GiB/s

omlettehead• 9 years ago

Sure, but you're building strings in memory. It might not be a lot of memory, especially if you're able to run Python, but the native `yes` command can run on the smallest of embedded systems, which is why its speed is impressive.

naranha2• 9 years ago

any idea why it is so much faster on fedora 26?

$ yes | pv -r > /dev/null

[10.3GiB/s]

totony2• 9 years ago

benchmarking is useless when comparing different conditions

ie your computer is probably faster than op's

naranha2• 9 years ago

Of course. Though 37MiB/s compared to 10Gi/s probably means that the GNU version used by Fedora has a faster implementation than Apple's version.

nurettin• 9 years ago

Rust strike force is back, rewriting Unix tools in a sane language

aleden2• 9 years ago

This topic comes up every now and then. I thought this post was particularly insightful,

"One thing to keep in mind when looking at GNU programs is that they're often intentionally written in an odd style to remove all questions of Unix copyright infringement at the time that they were written.

The long-standing advice when writing GNU utilities used to be that if the program you were replacing was optimized for minimizing CPU use, write yours to minimize memory use, or vice-versa. Or in this case, if the program was optimized for simplicity, optimize for throughput.

It would have been very easy for the nascent GNU project to unintentionally produce a line-by-line equivalent of BSD yes.c, which would have potentially landed them in the 80/90s equivalent of the Google v.s. Oracle case."

https://news.ycombinator.com/item?id=14543640

rurban2• 9 years ago

Not impressive at all. Basically he had to write a lot of manual buffering code to reach GNU yes throughput. I would suggest to use an infrastructure which already provides proper IO.

e.g. perl or clisp.

    $ perl -C0 -e'print "y\n" x (1024*8) while 1' | pv > /dev/null
    ^C.4GiB 0:00:11 [6.17GiB/s] [              <=>                                                     ]

    $ yes | pv > /dev/null
    ^C.3GiB 0:00:07 [6.64GiB/s] [         <=>

And with linux-only vmsplice the record stands at 123GB/s https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...

A Little Story About the `yes` Unix Command

Related Stories