sharkdp

Thank you!

I reported this as https://github.com/astral-sh/ty/issues/1994

Support for auto-completing TypedDict keys is tracked here: https://github.com/astral-sh/ty/issues/86

We should generally support TypeDicts. Can you go into more details of what is not working for you?

Is it possible that your program was waiting for input that never came?

We do close stdin to prevent this. So you can benchmark `cat`, for example, and it works just fine.

On the flip side (making things more random as opposed to less), something that randomizes the initial stack pointer would be nice, as I've sometimes seen this go really, really wrong (renaming a binary from foo to foo_new made it run >1% slower!).

This is something we do already. We set a `HYPERFINE_RANDOMIZED_ENVIRONMENT_OFFSET` environment variable with a random-length value: https://github.com/sharkdp/hyperfine/blob/87d77c861f1b6c761a...

“Best publicly available” != “great”

Of course. But it is free and open source. And everyone is invited to make it better.

Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions)

This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts

Please feel free to comment here if you think it should be included in hyperfine itself: https://github.com/sharkdp/hyperfine/issues/523

Automatic isolation to the greatest extent possible (given appropriate permissions)

This sounds interesting. Please feel free to open a ticket if you have any ideas.

Interleaved execution, in case something external changes mid-way.

Please see the discussion here: https://github.com/sharkdp/hyperfine/issues/21

It just… runs things N times and then does a naïve average/min/max?

While there is nothing wrong with computing average/min/max, this is not all hyperfine does. We also compute modified Z-scores to detect outliers. We use that to issue warnings, if we think the mean value is influenced by them. We also warn if the first run of a command took significantly longer than the rest of the runs and suggest counter-measures.

Depending on the benchmark I do, I tend to look at either the `min` or the `mean`. If I need something more fine-grained, I export the results and use the scripts referenced above.

At that rate, one could just as well use a shell script and eyeball the results.

Statistical analysis (which you can consider to be basic) is just one reason why I wrote hyperfine. The other reason is that I wanted to make benchmarking easy to use. I use warmup runs, preparation commands and parametrized benchmarks all the time. I also frequently use the Markdown export or the JSON export to generate graphs or histograms. This is my personal experience. If you are not interested in all of these features, you can obviously "just as well use a shell script".

If that's the case, I would consider it a bug. Please feel free to report it. In general, hyperfine should not take longer than ~3 seconds, unless the command itself takes > 300 ms second to run. In the latter case, we do a minimum of 10 runs by default. So if your program takes 3 min for a single iteration, it would take 30 min by default — yes. But this can be controlled using the `-m`/`--min-runs` option. You can also specify the exact amount of runs using `-r`/`--runs`, if you prefer that.

I could not find a good way to get Hyperfine to do what I wanted

This is all documented here: https://github.com/sharkdp/hyperfine/tree/master?tab=readme-... under "Basic benchmarks". The options to control the amount of runs are also listed in `hyperfine --help` and in the man page. Please let us know if you think we can improve the documentation / discovery of those options.

I don't think it's useless. You can use hyperfine to run multiple benchmarks at the same time, to get a comparison between multiple tools. So if you want it to work without quotes, you need to (1) come up with a way to separate commands and (2) come up with a way to distinguish hyperfine arguments from command arguments. It's doable, but it's also not a great UX if you have to write something like

    hyperfine -N -- ls "$dir" \; my_ls "$dir"

The issue is it runs a kajillion tests to try and be “statistical”.

If you see any reason for putting “statistical” in quotes, please let us know. hyperfine does not run a lot of tests, but it does try to find outliers in your measurements. This is really valuable in some cases. For example: we can detect when the first run of your program takes much longer than the rest of the runs. We can then show you a warning to let you know that you probably want to either use some warmup runs, or a "--prepare" command to clean (OS) caches if you want a cold-cache benchmark.

But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”.

What is the "best answer you can"?

It’s very much designed for nanosecond to low microsecond benchmarks.

Absolutely not. With hyperfine, you can not measure execution times in the "low microsecond" range, let alone nanosecond range. See also my other comment.

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond range. Maybe just below 1 ms if you take special care to run the benchmark on a quiet system. Everything below that (microsecond or nanosecond range) is something that you need to address with other forms of benchmarking.

But for everything in the right range (milliseconds, seconds, minutes or above), hyperfine is well suited.

Yes. If you don't make use of shell builtins/syntax, you can use hyperfine's `--shell=none`/`-N` option to disable the intermediate shell.

Caching is something that you almost always have to be aware of when benchmarking command line applications, even if the application itself has no caching behavior. Please see https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warm... on how to run either warm-cache benchmarks or cold-cache benchmarks.

365·243 ought to be 365·2425 exactly:

Yes. This is also how it is defined: https://github.com/sharkdp/numbat/blob/ba9e97b1fbf6353d24695...

The calculation above is showing a rounded result (6 significant digits by default).

Yes. If you want to know more, you can read about it here: https://github.com/sharkdp/numbat/blob/master/assets/reasons...

So 'year' refers to the Gregorian year and is equal to 365.243 days [1]. We also have 'julian_year' which is equal to '365.25 days'.

We also have 'sidereal_day' equal to '23.9345 hours', and if you believe it is useful, we can also add 'sidereal_years'.

[1] https://numbat.dev/doc/list-units.html

Frink is not open source, unfortunately.

Insect is the predecessor project. Numbat is the replacement. It also has an interactive version at https://numbat.dev/

Thank you for the reference. I hadn't seen Unchained.

You missed the point about this example though. I wanted to show how Numbat can help prevent the exact error that you made in your program. 'G_Newton * earth_mass * solar_mass / distance_sun' is not a force. It's an energy. How would you write type annotations in this Nim example to help you find that same mistake?

I looked at Julia and Unitful.jl quite a bit when designing Numbat. It looks great.

Extending it to cover all of Numbat's functionality could be trivially accomplished in a few lines.

Look, I'm not claiming that Numbat is superior. But I think Numbat might have its own little niche. And even if not, it's always good to have alternative solutions available.

- Numbat is specifically designed to handle physical dimensions and units. It has a special syntax to give developers the best experience when dealing with units. You can just type "KiB" and have it be parsed as kibibytes. You can use the "->" operator to convert to other units. You can directly annotate functions with physical dimensions ("fn kinetic_energy(m: Mass, v: Velocity) -> Energy = …"). Other languages and their unit libraries might have the same functionality, but it has always been added as an afterthought.

- Numbat has a static type system, Julia's is dynamic.

- Numbat can be compiled to Web Assembly and you can run it in your browser. I'm not sure if that is possible with Julia?

- Numbat might eventually be able to infer (physical dimension) function parameter types with its Hindley-Milner-style type system [1]. The only language that I know of that can do this is F#. The author of F#'s unit system wrote his PhD thesis on the subject [2], and Numbat follows the original approach in the thesis quite closely.

[1] https://github.com/sharkdp/numbat/issues/29

[2] https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-391.pdf

Yes. We use up-to-date currency exchange rates from the European Central Bank [1].

https://github.com/sharkdp/numbat/blob/786512175b99c195a7d5b...

Thank you.

I don't think we diverge from units on purpose anywhere, but I am probably not very familiar with its syntax, to be honest. I did however look at its huge collection of units and checked if there were any important units missing. I think we have a pretty comprehensive list of units that are supported by now in Numbat [1].

[1] https://numbat.dev/doc/list-units.html

No, we do not have aggregate types in Numbat yet. But it is definitely something I would like to support.

Note that it is possible to construct a type system solution to this problem (vectors/matrices with non-uniform units). A colleague of mine has an excellent talk on this: https://www.youtube.com/watch?v=SLSTS-EvOx4

By 'Scalar', in the document you referenced, we mean a dimensionless quantity. Not a scalar in the scalar-vector-matrix-tensor sense. Maybe I should rethink that notation.

F# is the only language I looked at that has a really powerful unit system. The author of that unit system wrote his PhD thesis on the subject [1]. Numbat's type system is actually pretty close to what was suggested in that thesis (in contrast to F#'s system, which is based on units — not dimensions).

[1] https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-391.pdf

As a physicist, if you ever see units on the parameter to a trigonmetric function, you can be fairly certain something is wrong.

Exactly. Which is why trigonometric functions in Numbat have the signature

  cos(x: Scalar) -> Scalar

Whether or not angles should be considered dimensionless is actually a matter of academic dispute. If you want to know more, take a look at https://github.com/sharkdp/numbat/pull/167 and the references therein.

I looked at Frink quite a bit and I agree that it looks cool. But as far as I can tell, it does not have a static type system. At least it's not based on physical dimensions.

And it is not open source.

You don't have to convert to scalars. Numbers have a type of 1 (= Scalar).

You can just call exp(3), for example: https://numbat.dev/?q=exp%283%29%E2%8F%8E

Or you could pass an angle quantity, which is convertible to a Scalar. Like cos(30 deg): https://numbat.dev/?q=exp%283%29%E2%8F%8Ecos%2830+deg%29%E2%...

I usually follow HN closely, but I would have probably missed it, if I hadn't received notifications from friends on three different channels. Thank you very much for writing the post!

Thank you for the kind words Martin. I hope we can continue this journey in the next couple of years :-)

Thanks Holger. I have fond memories of the BWInf times!

In the past years, I have actually experienced Hacktoberfest as a really great event - both as a contributor as well as a maintainer. The recent events sadden me, but I also understand why these changes were necessary.

I just opted in to Hacktoberfest PRs for two of my projects: bat and fd. For bat, I opened three special issues for Hacktoberfest in order to help contributors who are completely new to the open source world:

- https://github.com/sharkdp/bat/issues/1211

- https://github.com/sharkdp/bat/issues/1213

- https://github.com/sharkdp/bat/issues/1216

Curious to see how this works out. The first PRs are already coming in :-)