Don't Use Iperf3 on Windows

https://techcommunity.microsoft.com/t5/networking-blog/three-reasons-why-you-should-not-use-iperf3-on-windows/ba-p/4117876

by thepuppet33r • 2 years ago

76 50 2 years ago

techcommunity.microsoft.com

extraduder_ire• 2 years ago

I wonder how it performs under WSL. I assume worse than the suggested benchmarking tools, but far better than the outdated Iperf3 version using cygwin that the page references.

Dalewyn• 2 years ago

Given the goal is benchmarking and there are native options available, having emulation/thunking (WSL1) or virtualization (WSL2) in the pipeline would defeat the purpose unless you very specifically want such things.

TiredOfLife• 2 years ago

Even enabling wsl2 or other features puts windows itself into virtualized mode.

nullindividual2• 2 years ago

If you read the comments on the article, the author goes into WSL2 perf.

molticrystal• 2 years ago

Perhaps Microsoft should work with OpenWRT to get the linux version of ntttcp incorporated and built for routers as a package. I use iperf2/iperf3 because it is provided by opkg.

luma• 2 years ago

I think this is the key problem with their recommendation - it presumes that I'm running Windows on both sides of the test. iperf is neat because it runs on just about anything which allows you to run a binary which is really handy in datacenter troubleshooting situations.

pseudosavant• 2 years ago

My first thought was "I bet getting ntttcp on x86 with Ubuntu/CentOS is ok, but I doubt it will be simple for something like OpenWRT."

    root@router:~# opkg find ntttcp
    root@router:~# opkg install ntttcp
    Unknown package 'ntttcp'.
    Collected errors:
     * opkg_install_cmd: Cannot install package ntttcp.
    root@router:~#

Microsoft would be so much better served putting resources into proper Windows support for iperf3, instead of creating their own tools and convincing the world to switch.

rixthefox2• 2 years ago

I would suspect Microsoft's corporate culture is to blame for that. If you make a tool that "already works on Windows" work you don't get any extra praise. If you however come up with a brand new thing you'll get a slap on the back and a raise!

suprjami3• 2 years ago

There are only packages for Debian, Ubuntu, and NetBSD. Not even include-everything Arch or Gentoo has ntttcp apparently.

proactivesvcs• 2 years ago

"Go search for “iPerf3 on Windows” on the web. Go ahead, open a tab, and use your search engine of choice. Which I am certain is Bing with Copilot."

For many people it won't be by choice, it'll be because Windows' plethora of dark patterns and anti-consumer features have tricked you into using it, or taken away your previous explicit choice.

Joker_vD3• 2 years ago

I personally thought the part about Bing with Copilot was a subtle joke, because giving people the benefit of doubt is just a polite thing to do and even the Microsoft employees deserve it. Otherwise the sentiment contained in that sentence is indeed quite outrageous.

Anyhow, even when searching for "iPerf3 on Windows" in both Google and DuckDuckGo, the first result is still iperf.fr.

38• 2 years ago

Even with the most charitable Interpretation, it's a tasteless joke. Bing has a history of shoving Copilot down people's throats, and Microsoft has a LONG history of shoving Bing down people's throats. They built it into the God damn start menu, and have repeatedly changed the method to disable it. So even people who have explicitly disabled it will see it pop back up from time to time. Fuck Microsoft.

mrguyorama• 2 years ago

because giving people the benefit of doubt is just a polite thing to do and even the Microsoft employees deserve it

This is just woefully naive when we have court documents from Microsoft's history of open source attacks. Microsoft lost the benefit of the doubt ages ago, nevermind that you should not be giving a legal entity who's only incentive is to extract more wealth from the world "the benefit of the doubt".

PaulKeeble• 2 years ago

iperf3 is popular because its in the router software stack and a good way to check the raw performance. I know from my own experience that iperf3 on windows performs fairly similar to Linux on a modern machine all the way up to 10 gbit/s. Its a theoretical problem not a practical one but if Microsoft has a better solution then they can open source it and do the work to include it in the open source router software and it will get used.

nullindividual2• 2 years ago

If you read the article, the author shows iperf3 is slower than Microsoft's open source tools.

rixthefox2• 2 years ago

Nothing wrong if it works faster, but it's not impossible to make tools that are compatible with other standard testing programs. By intentionally ignoring the tools network engineers are already using every day Microsoft is really shooting themselves in the foot and showing that they are tone deaf to the needs of users outside of their walled garden.

How many routers do you know off the top of your head that run Windows?

supertrope• 2 years ago

Using v3.16 iperf3.exe I got 9.2 Gbps between Windows and Ubuntu. A single TCP connection. No -P flag needed.

klabb32• 2 years ago

The average [of iperf3] across multiple tests was about 7.5 Gbps.
Ntttcp averaged about 12.75 Gbps […] Ntttcp does something called pre-posting receives, which is unique to this tool.

This is TCP we’re talking about, and with large enough buffer—per-syscall (128k on my iperf3 by default - that’s good) you should be able to saturate even a thick line, provided auto-tuning is enabled correctly. But just add -P 16 to parallelize to be sure it’s not the TCP stack, instead? Did they even try a workaround, before suggesting an extremely limited and proprietary tool that nobody is familiar with? Or better yet, help the project out by contributing native code if the open source community is so bad at windows? At the very least, make a wire-compatible version.

Iperf is a very boring and wonderful tool that just works. Importantly, it’s a networking tool, which means you need interop across different systems. Any networking tool that’s single-platform is dead in the water, as far as I’m concerned.

Edit: To be fair, I see they have a Linux version, how generous. Last updated in 2022 and many open issues though.

distortedsignal• 2 years ago

After reading the article, it sounds like iPerf3 on Windows is going through an emulation layer (Cygwin) which - I kinda get why it's going slower if that's the case. Isn't syscall translation always going to add overhead?

raggi2• 2 years ago

It’s a shame that they don’t talk about it in the article but a big part of the drive toward next generation tools is to study the interfaces needed and used by quic implementations which can’t “just be solved by” the flags you’re talking about, and iperf has no support for the relevant features (offloads).

klabb32• 2 years ago

Yeah, to be fair I’m assuming tcp both on the iperf side (which I always use it for) and the alternative ntttcp tool does have TCP in the name although it appears they have UDP support as well(?)

I know UDP and QUIC comes with a ton of novel bottlenecks throughout the stack (even down to the NICs, no?), and at the very least tons of knobs, which warrants more custom tooling and keeping up with the latest in terms of OS/platform support. I can totally see how iperf isn’t suited for that.

Still, it’s a pretty strong and different message that Microsoft is sending on an official channel, which is unfortunate if it indeed is the case that iperf works just fine on windows with pretty standard flags (note that even on typical Linux recv buffers are constrained to ~6MiB so -P is standard for WAN saturation tests).

(Also good to see your relentless attention to engineering details as always. We used to work together some years back)

SushiHippie2• 2 years ago

This article is actually right that I downloaded the old version for Windows (would not happen on another OS, as there are maintained packages in most of the package managers).

But I just checked with the newer version and I didn't see a difference when testing with my 2.5GbE LAN (still 2.3Gbit UP and 1.9Gbit DOWN)

jtriangle2• 2 years ago

Just an aside, when I'm forced to use windows, installing programs with choco is a much better experience.

It's not quite linux package management, but, it beats the pants off any other way.

darknavi• 2 years ago

I am a big fan of winget these days.

Joker_vD3• 2 years ago

Do you really want Microsoft to maintain a centralized software repository from which you, yes, you, are supposed to install almost all of the software you'd like to use? Do you, really?

It's always such a bizarre experience when people advocate for centralized repositories for Windows. Windows, unlike Linux, comes in very few flavours/versions (and with decent backwards-compatibility story), so the application developers can very well be (and in practice, well, are) expected to build the apps for the Windows versions they care about and then use their web-sites to distribute it.

In this case with iPerf3 we just have a sad story of two forks, and the group that owns the namesaked website for some reason could not be bothered after 2016 to re-upload the releases of the other group. Well, similar things happens in "official" Linux distros as well, with severely outdated packages.

zamadatix• 2 years ago

I, really, wouldn't mind installing most of my software from Microsoft maintained repositories. The kinds of things that drive me nuts are when these are the ONLY place you can get things and you can't just add other repos like you can on any Linux setup. Managing distro variations doesn't really come into why I like to use centralized repositories.

If winget had appeared a decade or two earlier it might have replaced the need for every Windows app to ship an auto-updater.

mschuster91• 2 years ago

Do you really want Microsoft to maintain a centralized software repository from which you, yes, you, are supposed to install almost all of the software you'd like to use? Do you, really?

I'd actually love for that, yes. Just google "download VLC" - the top link does refer to VLCs official site (probably because they paid for or got donated the top spot), the second one goes to an IT newspaper, the third one to something uBlock flags as badware risk, the fourth one another newspaper, the fifth one someone offering VLC with a newsletter subscription, and finally the official page.

And that's the reality for a lot of popular Windows software. Part of that is obviously the responsibility of Google that can't even keep the top software search results free of advertising grifters (the newspapers) or malware, but the larger part is on Microsoft for not offering a centralized store for decades.

In contrast, on Ubuntu a simple `sudo apt-get install vlc` is enough, and (bar a supply chain corruption) I can be reasonably sure that I get a stable, unmodified version of VLC. Oh, and when I run a `sudo apt-get remove --purge vlc` it is gone... in contrast to a lot of Windows software with badly written uninstallers (because MSI is an utter PITA to develop against), that leaves remains everywhere across the system.

drbawb• 2 years ago

I was able to get up to 35-37 Gbps between my host-guest with a virtio NIC with the outdated iperf3.exe, which I did indeed acquire from the site mentioned in the article. That seems like plenty of perf, to me, considering the fattest link I could possibly have in my house would be ~35Gbps if I aggregated all the ports on my switch. (... and filled my Threadripper system with nothing but NICs, I guess?)

I actually get ~7 Gbps, not 10Gbps, out to my real network. I haven't dug into what the issue is but when hitting my real network the guest clearly becomes bottlenecked by CPU, so I doubt iperf3 is to blame. (The host does not have this problem despite being on the same bridge, so I'm guessing there is some host-guest optimization I'm hitting in the virtio driver in the former case.)

Now that's a low-latency, high-bandwidth link. If you're testing "high latency, high-bandwidth" as purported in the article, and that link is apparently ~40Gbps or fatter, you're probably running on a "real server" in a "real datacenter" somewhere. I can tell you I wouldn't be burning a Windows Server license just to verify my L2/L3 connectivity is configured correctly.

I am sure MS would love if I bought two licenses of their proprietary operating system just to use this proprietary network testing client, but my pockets are not infinitely deep. (As I already spent all that money on the Cisco-branded optics. /s)

banish-m42• 2 years ago

Windows Server is barely used anywhere anymore outside of Azure and niche IT operations that demand it. CentOS Linux, Amazon Linux, and Debian Linux run on 100's millions to a billion boxes.

jiggawatts• 2 years ago

Over the years I've used a variety of network test tools including ntttcp, iperf, hrPing, and I even whipped a couple of custom ones.

I've gravitated towards using custom test tools that use the same pattern as the application that will run on the network. So instead of testing maximum UDP throughput or with dozens of TCP threads with artificially high window sizes, I test with "whatever it is" that the client/server apps are likely to do.

I still use hrPing, because it behaves a lot like a large category of common server apps, such as Linux tools ported to Windows.

eqvinox2• 2 years ago

Good recommendation if you need to test for some target application's performance.

However, when the service you provide is the network itself, e.g. to a customer who just bought XYZ performance from A to B, you need usage-agnostic test tools.

suprjami3• 2 years ago

The problem there is commercial users want trusted software from their support providers.

A company providing a network application should be providing a custom benchmark tool whose behaviour mimics the application. If the application provider is so incompetent they don't do that, then tools like yours become useful, if they are allowed by organisation security policy. In my experience, few are going to compile some bloke's random testing tool from source or run scripts off your GitHub.

Most commercial users just want to plug in 10Gbps NICs, run iperf and see "9.x Gbps", tick a box and move onto the next item.

eqvinox2• 2 years ago

What Does Microsoft Recommend
Microsoft maintains two synthetic network benchmarking tools, ntttcp (Windows NT Test TCP) and ctsTraffic.

Okay, let's see which of these is compatible with the iperf3 available on my locked-down commercial router (or, alternatively, the test endpoint my ISP provides) …

…it's neither…

…aaaaand we're back to iperf3 on Windows. :-(

As tongue-in-cheek this comment may be, a recommendation for 2 incompatible tools is worth nothing. Iperf is the de-facto standard, if anything the problem is sometimes you have iperf2 around for some reason. If, Microsoft, you want better Windows network stack benchmark results, you'll need to make something compatible with iperf3 (or fix iperf3.)

(It does not matter that ntttcp runs on Linux. I can't install ntttcp on an Ubiquiti router, or ask my ISP to add another test endpoint.)

jandrese• 2 years ago

Yet again this article shows how Microsoft programmers don't dogfood their own console applications.

Compare invocations for these commandline apps:

Linux app:

    iperf3 -s

Windows app:

    .\ctstraffic.exe -listen:* -Buffer:"$(128KB)" -Transfer:"$(1TB)" -ServerExitLimit:1 -consoleverbosity:1 -TimeLimit:60000

Who the hell wants to type all of that out?

banish-m42• 2 years ago

Steve Jobs might've been an opinionated dick, but he was right that Microsoft "has no taste in a big way". CLI UX with sane defaults and shorter options wins.

cchance• 2 years ago

are really none of those defaulted? i kinda fiugred that example was just setting it's defaults to match the iperf defaults

suprjami3• 2 years ago

ntttcp only has packages for Debian, Ubuntu, and NetBSD:

https://pkgs.org/download/nttcp

No user of the other commercial Linux distros (RHEL, SLES) is going to install ntttcp from source or random binaries.

Of course, the Windows kernel team could support POSIX system calls that iperf and other network applications require, instead of removing that feature altogether:

https://learn.microsoft.com/en-us/archive/technet-wiki/10224...

But hey why provide customers a good product when you can just write a blog with a useless recommendation instead.

Microsoft is a big company with many teams on many products. Some of those teams really understand interoperability with modern developers and applications. This team don't.

pjmlp• 2 years ago

Actually had Windows NT provided better POSIX support, I would never ever had bothered with dual boot into BSD or Linux during the 1990's.

In hindsight not doing that is what opened the door to FOOS on PCs in first place, so beware what one wishes for.

rjmcmahon2• 2 years ago

Just some clarification: IPerf 2 is different from the iperf3 found at https://github.com/esnet/iperf Each can be used to measure network performance, however, they DO NOT interoperate. They are completely independent implementations with different strengths, different options, and different capabilities. Both are under active development.

The current release of iperf 2 is 2.2.0 but 2.2.1 with bug fixes will be out soon.

Comparison table here: https://iperf2.sourceforge.io/IperfCompare.html

rjmcmahon2• 2 years ago

Also, with iperf 2, -c localhost seems to perform the same on Windows as it does for Linux. So skeptical that the cygwin overhead is very large.

A binary is here https://sourceforge.net/projects/iperf2/files/

Don't Use Iperf3 on Windows

Related Stories