24-core CPU and I can’t move my mouse

https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and-i-cant-move-my-mouse/

by joebaf • 9 years ago

1011 492 9 years ago

randomascii.wordpress.com

The terrifying result of having ABI compatibility with the first commercially successful system of its kind from the '80s.

Granted, they aren't doing themselves any favours with their new straight-jacket style application ABI.

willvarfar2• 9 years ago

You are trotting out the same tired line without reading the article carefully? ;)

This is a regression. The article shows it worked fine up to Windows 7.

Aeolun• 9 years ago

7 is the best

jbg_• 9 years ago

How is ABI compatibility with early systems the cause of this bug? You'll note that it wasn't present in Windows 7.

LeoPanthera• 9 years ago

64-bit versions of Windows 10 (and 8/7 maybe?) are no longer compatible with 16-bit code.

ripdog• 9 years ago

Yep, no 64-bit Windows has 16-bit compatibility.

yuhong5• 9 years ago

I can't imagine that moving the Win32k stuff back to CSRSS would help much in this case, right? Though it is still a good thing especially for terminal servers where hopefully one CSRSS process crashing just terminate the session.

sjmulder• 9 years ago

moving the Win32k stuff back to CSRSS

Out of curiosity, was there some big move of functionality from CSRSS into Win32k earlier? When and what?

yuhong5• 9 years ago

In NT4.

isatty• 9 years ago

I read mouse as house and was confused for the longest time.

mjsweet• 9 years ago

Same happened to me... initially I thought it must be some 1st world problem.

aramadia• 9 years ago

When I saw his workstation specs, I thought, thats the exact same one I have at work! Then I checked the bottom, yep, he's at Google too.

unixhero• 9 years ago

Lucky bastards

willvarfar2• 9 years ago

Does yours run Windows? Do you get to choose? If you could choose, which OS would you use? Any clear favourites with your peers for development? Just collecting anecdotes ;)

albertzeyer• 9 years ago

I was wondering what ETW is, and found in one of his other posts: ETW (Event Tracing for Windows).

He seems to be one of the main contributors of https://github.com/google/UIforETW

Seems to be quite useful.

---

About the post: tl;dr: NtGdiCloseProcess has a system-wide global lock which is used quite often e.g. during a build of Chrome which spawns a lot of processes. This problem seems to be introduced between Windows 7 and Windows 10.

I thought that there would be a solution or a fix but it seems this is not yet fixed. "This problem has been reported to Microsoft and they are investigating."

yuhong5• 9 years ago

I wonder if this is the same lock most win32k call takes.

masklinn3• 9 years ago

NtGdiCloseProcess has a system-wide global lock which is used quite often

"Holds" rather than "has", and importantly that system-wide lock must be held by things like SendMessageW (which sends a message and waits for its processing before returning) which is pretty critical for UI updates.

This is compounded by parallelised build processes as not only do they spawn lots of processes leading to lots of process destruction: it both fucks up the UI and completely serialises process destruction so your nice 24 process flights for faster processing end up taking multiple seconds to all shut down, and the more cores (and thus processes) you have the worse it is, and the worse your stutters.

qb454• 9 years ago

"Holds" rather than "has", and importantly that system-wide lock must be held by things like SendMessageW (which sends a message and waits for its processing before returning) which is pretty critical for UI updates.

Do you happen to know anything about it? I'm scratching my head how it's possible that process termination serializes with GUI... Maybe it hogs some lock on process descriptors which SendMessage also needs to grab for a moment to find the target process? I hope you didn't mean to say that every SendMessage call is completely serialized with each other.

mhw• 9 years ago

As a Windows outsider, I'm puzzled why programs used as part of the Chrome build system (which I'd expect to only use console I/O) are using APIs that cause interactions with the GUI? By analogy, is this not like gcc redundantly setting up a connection to the Xserver each time it is run?

(I'm making an assumption that NtGdiCloseProcess is part of the GUI API (GDI == Graphics Device Interface) which is why it may interact with the GUI message passing.)

yuhong5• 9 years ago

More precisely, I believe the kernel tracks what threads are "GUI" threads that use win32k.

slededit• 9 years ago

In the very early days of NT the GDI subsystem was in userland and you wouldn't have this problem. Unfortunately it was too slow for machines of the 90s and so GDI+user32 is very tightly integrated with the kernel.

Even to the point where it does neat things like callback user mode code from the kernel. Unwinding this without breaking things is nigh impossible at this point.

weinzierl• 9 years ago

I grew up on the Commodore 64 (1 Core, 1 hyper-thread :-), almost 1 MHz clock freq, almost 64 K usable RAM).

The machine was usually pretty responsive, but when I typed too quickly in my word processor it sometimes got stuck and ate a few characters. I used to think: "If computers were only fast enough so I could type without interruption...". If you'd asked me back then for a top ten list what I wished computers could do, this would certainly have been on the list.

Now, 30 years later, whenever my cursor gets stuck I like to think:

"If computers were only fast enough so I could type without interruption..."

fredley• 9 years ago

It's all just trade offs. However fast or powerful your machine is, software will use as much of that resource as possible, up to the point where it occasionally interferes with input (but not too much or you'll switch to something else).

anigbrowl5• 9 years ago

But why? What good is an operating system on a multi-core device that allows anything to get that close to the performance envelope? This is a fine example of competition driving change for change's sake rather than real innovation and everything ending up worse as a result. I like new features as much as the next person, but not when they compromise core functionality. Not being able to type is inexcusable.

agumonkey2• 9 years ago

There's a talk by Guy Steele 'growing a language', he says that ideally a language should shrink as it gets smarter semantics.

gkya• 9 years ago

That's is better rewritten as "however fast or powerful your machine is, software will waste as much of that resource as possible".

Aloha• 9 years ago

Similar in sentiment to "I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone."

ziotom78• 9 years ago

I'll definitely steal this!

Yuioup• 9 years ago

Your Commodore 64 had one hyper-thread? Wow, that's forward thinking.

torrent-of-ions• 9 years ago

Plus ça change, plus c'est la même chose...

jmartinpetersen• 9 years ago

"The more it changes, the more it’s the same thing." [0]

[0] https://en.wiktionary.org/wiki/plus_%C3%A7a_change,_plus_c%2...

anigbrowl5• 9 years ago

I have almost the same background (Vic-20, if you thought programming was hard try having only 3.5k of RAM) and feel the same irritation...but I don't wonder why computers aren't faster, I wonder why operating systems and application is so utterly shit.

I have a Moto X phone (not too old) and watching it slow down is almost comical. Sometimes I'll decide to close a few apps and hitting the selector button brings up screenshots of all the open apps. Then I have to wait a few seconds for the X to appear on the app frames so I can close them.

If I had the inclination and temperment to be one of those YouTube personalities who makes a schtick out of complaining about things I would have no shortage of material for a 'designed by idiots' channel.

staofbur5• 9 years ago

I think most of the shit is concentrated on a couple of notable platforms: Windows and Android. Outside of that space I've noticed less productivity hampering nightmare tools and OS features. Really I get perhaps one or two bits of stupid from Linux a year on the server-side, usually when integrating it with Windows ironically, and on the macOS/iOS front I haven't had a notable issue since I switched about a year ago.

floatboth2• 9 years ago

You don't have to wait for the X. Just swipe them away.

Or just don't close activities ever, it's pretty pointless. It doesn't actually kill the app if it's still open.

RJIb8RBYxzAMX9u• 9 years ago

Except without interrupts, your computer would miss all the keystrokes. :-)

hypervis0r2• 9 years ago

That's so deep, it hit me in the IDT

agumonkey2• 9 years ago

way more naive but my first own desktop was a P75, a few years down a friend of a friend got his hand on a batch of P233MMX and I started to fantasize that these could be so fast, the menus may appear before you even finished clicking. I didn't know how non linear computers were at the time.

Few things also, I'm often surprised how fixed is the responsiveness of systems, hardware grew 3-4 orders of magnitude but the cruft and chaos plus the change of resolutions keep latency around the same value. It's sometimes even worse. Psychologically when I boot a 64MB windows machine and can enjoy word / IE / winamp I feel very weird (video could be offloaded to a chromecast for that matter).

twobyfour3• 9 years ago

The worst is when this happens when you're not doing anything that should be computationally intensive, just entering text in a web app that abuses JavaScript.

Houshalter• 9 years ago

Even just entering text into the search bar uses loads of resources. Every character entered does a full search of your history and bookmarks, sends a request to Google to do autocomplete, prefetches the web page you are typing, applies spellcheck, etc.

digi_owl4• 9 years ago

Why i refuse to indulge the latest eyecandy and bling from the desktop world if possible.

If i didn't need a GPU to do window switching back then, why do i need it now?

swiley4• 9 years ago

FVWM and VTWM switch windows and workspaces instantaneously on my crappy netbook with just the EFI framebuffer.

I always laugh a little inside watching people try to do the same on windows 10 and OSX with all the hardware acceleration and waiting multiple for seconds.

marcosdumay3• 9 years ago

To save battery power.

mmargerum• 9 years ago

I remember writing C code on my Commodore 64 in my early teens using Power C from spinnaker software. If memory serves I had to put 3 different 5 1/4 disks in to compile / link. There were compiler 1, compiler 2, and a linker diskette.

ghostly_s• 9 years ago

Weird that no-one here is mentioning how Windows is the only platform that still has this problem.

UI threads on a graphical desktop should always be the most privileged processes.

alacombe2• 9 years ago

So what now ?

Should/does the author wait getting some traction on Hacker News and hopefully be noticed by dev at Microsoft, or is there some way to provide Microsoft directly these data, skipping Tier-1 support ?

The author seem to be working at Google, so he might get some leverage from that, but what about Random Joe ? Keep enjoying the bug "forever", I guess ?

jsnell• 9 years ago

From the article:

This problem has been reported to Microsoft and they are investigating.

BinaryIdiot• 9 years ago

Did you read the article? The bug was already submitted to Microsoft and they are investigating.

So the answer to your question is: wait until they're done investigation and possibly release a patch.

alacombe2• 9 years ago

I skipped the conclusion, though, I'd still be interested about the channel that could be used to report such issues.

ximeng2• 9 years ago

There are many many MS bugs that last for years. Pretty much any time you search for a bug in MS software you will find someone in a support forum giving generic advice (known issue that we will work on, not a bug, please reinstall Windows and applications) and users complaining.

Typical example: https://excel.uservoice.com/forums/304921-excel-for-windows-... basic problem that has been there many many years. For this one there are workarounds but many others there are not.

The only real long-term solution is supporting competitive products.

intoverflow2• 9 years ago

The only real long-term solution is supporting competitive products.

Not really viable when the only OS that has software parity is OS X and the only OS with hardware parity is Linux.

If we drew a venn diagram it would be a straight line of circles with windows sitting in the middle. To switch away you have to make a choice between less software or less processing/GPU power.

puzzle• 9 years ago

The author also used to work at Microsoft in the performance team...

eecc2• 9 years ago

so basically a fork-bomb. I think Linux can still buckle under one, nothing that obscene...

mattnewton• 9 years ago

No, the article says this is a regression in how processes close.

masklinn3• 9 years ago

A fork bomb exhausts all system resources, this is not that, it doesn't actually use any resources, rather it locks the system out of them by serialising both process termination and UI updates.

eecc2• 9 years ago

Well, a fork bomb doesn't really use anything except it makes the process table management extremely time consuming. Not quite the same, but still conceptually similar

yanpanlau• 9 years ago

Just use linux

jlebrech• 9 years ago

that's asking for downvotes, I voted this up nonetheless.

gurkendoktor2• 9 years ago

Staying responsive (under load) is also still a work in progress on Linux:

https://www.phoronix.com/scan.php?page=news_item&px=BFQ-Queu...

dispat0r• 9 years ago

That is a disk scheduler and mostly for spinning disks. I haven't seen a stuck mouse cursor for a long time in linux.

SigSegOwl• 9 years ago

Thats why my machine gets that slow after running for weeks... explains everything!

blunte• 9 years ago

This is great work. I hope MS can improve Windows 10 by fixing this. I just added a new Win10 laptop with much better specs than my 3 year old rMBP, and I'm shocked by how much apparently random latency I experience with the UI in Windows 10 compared to the Mac. That's not to mention the issues of sloppier track pad (which constantly detects my left hand while I type) or the ungodly slow unzip (via 7z).

If only Apple would give us more than 16GB of RAM (in a laptop)... what a frustrating world for developers.

ShinTakuya• 9 years ago

I mean the trackpad issue is generally down to the manufacturer and the drivers they provide, but I take your point otherwise.

A pro tip with 7z is to use the two pane interface to extract and not dragging files to Explorer. The latter option extracts the files to a temp directory before copying them whereas the former extracts directly to the destination.

The Windows file systems are pretty cruddy in general though so decompressing tonnes of small files in general will take longer than on OS X or Linux.

SyneRyder• 9 years ago

I mean the trackpad issue is generally down to the manufacturer and the drivers they provide...

Was just wondering this myself. I tried a Surface Laptop this week, and its trackpad has everything I love from the 2012-era Mac trackpads. A satisfyingly snappy physical mouse click, gestures & acceleration, and a perfect size. (I can't stand Force Touch on the new MacBook Pro, and I don't like their new giant trackpads.)

deelowe• 9 years ago

A pro tip with 7z is to use the two pane interface to extract and not dragging files to Explorer. The latter option extracts the files to a temp directory before copying them whereas the former extracts directly to the destination.

That's insane. I had no idea. Thanks for the tip!

StillBored3• 9 years ago

  The Windows file systems are pretty cruddy in general though so decompressing tonnes of small files in general will take longer than on OS X or Linux.

I haven't checked this in a while, so it could be out of date... But what you claim is windows being cruddy, might be a symtom of a slightly different mindset in data protection. Here is a test, grab a big tar.gz/zip file and uncompress it in linux and in windows from the command prompt. As soon as you see it complete, pull the power plug/battery from the machine. Plug it back in, check the result.

In the past windows seemed to be more aggressive about flushing things to disk, and applications were far more generous with calls to FlushFileBuffers() on windows than similar apps on linux were about calling fsync(). Seemingly because fsync() is a machine deathblow on linux when under IO load. Where as windows did a slightly better job of only flushing the requested file. Also, i've seen a number of "streaming" apps on windows play nice and use FILE_FLAG_WRITE_THROUGH.. which makes them run slower, but doesn't trash the whole system.

Bottom line is that I wouldn't call NTFS cruddy, it does a lot of things that still aren't mainstream in linux filesystems. If your seeing some huge performance anomaly between windows & linux look closer. NTFS may not be sexy, but its a solid piece of engineering (ok some things are cool like VSS. TxF is also cool but might get deprecated because no one knows about it).

thriftwy2• 9 years ago

I have just bought a Windows 10 laptop for my wife.

I don't know if it is random updates or junkware or whatever, but sometimes the system will lock out for minutes - task manager shows <50% of load on any of disk, ram and cpu, but even waiting for task manager to launch can take a minute! I've witnessed a simple program (FAR manager) taking a minute to show its UI!

xaa• 9 years ago

Yeah, I've been out of the Windows game for some years now, but my wife has a few (desktop and laptop), occasionally comes to me and says "it's slow. fix it." And indeed it is slow, even on moderate browser use -- things just hang. Specs are more than adequate.

I have to say (after halfheartedly looking into it a few times): "Um, sorry honey, I have no idea. Guess it's a Windows thing." She nods, and goes back to patiently waiting minutes for the browser to stop hanging! Seems that people using Windows just accept this kind of thing as a given, to the point they don't even notice it. (I've installed Linux for her a few times, but of course, it's too different, doesn't run Photoshop, etc...)

SolarNet3• 9 years ago

Try turning off all of the malware and search features. Windows defender will run on every file, folder, or executable you touch. Play a movie with VLC? Here let us scan VLC, every DLL it depends on, every file you have opened with it, and the entire file you are about to open. They also bundled Cortana and Bing with search, which means you can get ads in your search! But also means it makes like twenty web queries every time you search. Also search indexes like non-stop for no reason I can fathom, which is a shame because I actually liked the windows 7 search features (which are now bundled with this ad serving, resource hogging, shit).

Also, don't trust task manager, it lies about utilization if you aren't running as admin and don't have it configured correctly.

dm319• 9 years ago

The latency and unzipping problems can be solved with linux. Though the trackpad could get worse. The random latency and unexplained HD activity always offended me in Windows - it is _my_ laptop after-all!

sgift• 9 years ago

Probably not really feasible, but I'd be interested if something comparable happens when your build process uses many threads instead of many processes. I still think using processes instead of threads is a hack, though I know the mainstream opinion says nowadays processes are the way to go and threads are a hack.

titzer2• 9 years ago

Full disclosure: I work for Google on Chrome.

A Chrome build is truly a computational load to be reckoned with. Without the distributed build, a from-scratch build of Chrome will take at least 30 minutes on a Macbook Pro--maybe an hour(!). TBH I don't remember toughing out a full build without resorting to goma. Even on a hefty workstation, a full build is a go-for-lunch kind of interruption. It will absolutely own a machine.

How did we get here? Well, C++ and its stupid O(n^2) compilation complexity. As an application grows, the number of header files grows because, as any sane and far-thinking programmer would do, we split the complexity up over multiple header files, factor it into modules, and try to create encapsulation with getter/setters. However, to actually have the C++ compiler do inlining at compile time (LTO be damned), we have to put the definitions of inline functions into header files, which greatly increases their size and processing time. Moreover, because the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file! Don't even get me started on templates. Oh, and at the end of the day, the linker has to clean up the whole mess, discarding the vast majority of the compiler's output due to so many duplicated functions. And this blowup can be huge. A debug build of V8, which is just a small subsystem of Chrome, will generate about 1.4GB of .o files which link to a 75MB .so file and 1.2MB startup executable--that's a 18x blowup.

Ugh. I've worked with a lot of build systems over the years, including Google's internal build system open sourced as Bazel. While these systems have scaled C++ development far further than ever thought possible, and are remarkable in the engineering achievement therein, we just need to step back once in a while and ask ourselves:

Damn, are we doing this wrong?

epistasis• 9 years ago

Yes, we are doing this wrong! But we don't have any right options yet either. I think another way of thinking about this is that if we can't imagine a better system, then the field is so mature that it has become solved and completely boring. I think systems programming is still a field for which there are new and novel problems to solve, or at least better designs to implement.

djhworld• 9 years ago

Wasn't this one of the reasons why the Go project started at Google, not necessarily for Chrome, but because C++ project compile speeds were horrendous for some internal projects.

Not saying Chrome could and should just switch to Go, it definitely would not be the right fit! But it's interesting that these sorts of builds still occur and consume a lot of developer's time.

jdcarter• 9 years ago

Yes, C++ build times were a significant factor in Go's creation. Rob Pike describes it here:

https://talks.golang.org/2012/splash.article

He covers all the same points about header files and how Go addresses those issues.

blt• 9 years ago

The modules proposal for C++ has been around a long time. We need it now more than ever. Projects are getting bigger and the C++ culture is evolving from runtime polymorphism to compile time polymorphism. Text inclusion is just not good enough.

vidarh4• 9 years ago

Damn, are we doing this wrong?

Yes: Not isolating different modules sufficiently to allow you to avoid including most headers when compiling most modules.

Patterns to do this in C++ has been well understood for two decades:

Strict separation of concerns coupled with facades at the boundaries that let all the implementation details of the modules remain hidden.

Yes, it has a cost: You're incurring extra call overhead across module boundaries, and lose in-lining across module boundaries, so you need to choose how you separate your code carefully. But the end-result is so much more pleasant to work with.

oblio2• 9 years ago

While what you're saying might be true, I can't help but think about Pascal units and say: "It didn't have to be this hard! We solved this problem 40 years ago!"

lucian1900• 9 years ago

Unfortunately that's not feasible for highly performance-sensitive projects like browsers or games.

bluedino• 9 years ago

Have you worked with Firefox? How does the build compare?

robin_reala2• 9 years ago

Firefox is C and Rust, so I’m not sure there’s such a direct comparison.

taway_1212• 9 years ago

AFAIR, the full build was also around 30 minutes on a desktop i7. Unfortunately, incremental build (even for the most tiny changes) was around 2-3 minutes which was a real killer for me when I tried to do some open source work on it.

pjc502• 9 years ago

"Concatenated" builds seem to be the best band-aid for this. Concatenate as many source files as possible before compiling, #include a bunch of cpp files into one big file. It makes tracking down errors slightly harder and macros a bit more risky, but greatly improves the overall build efficiency.

ori_b2• 9 years ago

And trashes incremental compilation time, which is what really matters.

jhasse5• 9 years ago

Also called unity build sometimes.

faragon2• 9 years ago

Are you sure C++ is the only thing to blame, and not how you use it?

IshKebab• 9 years ago

Is this the C++ version of "you're holding it wrong"?

egeozcan• 9 years ago

Is there any project that compiles significantly faster than Chrome while being in the same league in terms of LOC? I'd like to know if you have seen such an accomplishment even if its source is closed.

lmm• 9 years ago

Move to a language with a real module system?

claudiug• 9 years ago

Have you give a look to dlang? or even rust

jhasse5• 9 years ago

Rust's compile times are even worse.

miltondts• 9 years ago

I would like to see a more systematic solution to this problem. Most computers don't have a way to choose between throughput and real-time. So, I would love to see this available and preferably exposed as a run-time toggle.

rurban• 9 years ago

just use zapcc, and compile-time will be down. it's basically clang-4.0 so it's safe to use.

bonzini2• 9 years ago

If it's basically clang 4.0, why is it not part of clang? Projects like zapcc remind me of why the GPL is a good thing.

(I predict that in 10 years all the various soft cores will be variants of RISC-V each with its own old, unmaintained and proprietary fork of LLVM).

mpweiher5• 9 years ago

Damn, are we doing this wrong?

Yes, and I mean "we" as in "this industry".

I just recently talked to someone whose Swift framework(s) were compiling at roughly 16 lines per second. Spurred on by Jonathan Blow's musings on compile times for Jai[1], I started tinkering with tcc a little. It compiled a generated 200KLOC file (lots of small functions) in around 200 ms.

Then there are Smalltalk and Lisp systems that are always up and pretty much only ever compile the current method.

We also used to have true separate compilation, but that appears to be falling out of favor.

Of course none of these are C++ and they also don't optimize as well etc. Yet, how much of the code really needs to be optimized that well? And how much of that code needs to be recompiled that much?

So we know how to solve this in principle, we just aren't putting the pieces together.

[1] https://www.youtube.com/watch?v=14zlJ98gJKA He mentioned that a decent size game should compile either instantly or in a couple of seconds

pjmlp23• 9 years ago

Bjarne's made the mistake of having C++ rely on C's linker model, so it meant no modules.

Now we can see it as a big mistake, but on those days probably it was one of the reasons why C++'s adoption took off.

Also while C lacked modules, most Algol and PL/I derived languages supported them since the late 60's.

Swift's case has the issue of mixing type inference with subtyping, so lots of time is spent there.

All in all, I really miss TP compile times and at least on Java/.NET, even with AOT compilers are close enough.

EDIT: some typos

JabavuAdams• 9 years ago

I used to be a C++ guy for 20 years, but won't go back unless absolutely necessary. I mean, it's tempting -- there are some cool new language features. I find that the abstractions are leaky, though, so you still have to understand all the hairy edge-cases. The language has gone insane. I'm 40 -- I want to get stuff done before I die, not play clever games to get around my language / system.

jchw2• 9 years ago

Go is a good example of a language with fast compilation times. Of course, the optimizer needs improvements, but I believe they keep managing to make it faster as they improve the output rather than slower.

andrepd5• 9 years ago

Are you serious? You want to make a product: a Web browser. What technology are you going to choose? The one that makes your browser fast but gives you more work, or the one that makes your browser slower but makes compilation less of a nuisance to you? It's mind boggling that some would openly say that, hey, who cares about performance that much, these compilation times bother me, the developer. On a browser of all things! Would you be okay with your browser being 2x or 3x slower?

qznc2• 9 years ago

I'm not sure if the approach of Lisp and Jai (and also D) is the best one. They all have practically arbitrary code execution at compile time, so they can be arbitrarily slow to compile. In C++ template-programming is so hard that few people do it, but in those languages it is just as easy as normal code.

With mainstream languages, code generation is done by the build system which can avoid repetition. Caching generated code feels like a good idea to me. Doing it with compile time execution is (unnecessarily?) hard.

mtanski• 9 years ago

This whole sub-thread tangent-ed into C++ is bad because it's compilation causes this problem. Problems with C++ builds are well understood...

This specific issue prob impact other batch workloads with lots of small tasks (processes). There's no reason this should be happening on a 24 core machine.

dmead2• 9 years ago

what do you think would be a better choice than c++ at this point? clearly it hasn't helped with memory consumption. is mozilla getting it right-ish with rust?

also, would it kill you to like, coordinate something with the gmail team so their page doesn't kill my machine after being open for a couple of house?

come on guys.

nneonneo2• 9 years ago

These numbers don't seem abnormal; I recall building Safari many years ago, and having multi-GB of intermediate products shrink down to a 30MB executable (plus a few hundred MB of debug symbol).

So, I have a thought: if we're spending all this time to compile functions (particularly template functions) that are just thrown away later, why are we performing all our optimization passes up-front? Surely, optimization passes in a project like Chrome must eat up a lot of compilation cycles, and if that's literally wasted, why do it in the first place? Can we have a prelink step where we figure out which symbols will eventually make it, and feed that backwards into the compiler?

Maybe a more efficient general approach might be to simply have the optimizer be a thing that runs after the linker, so that the front-end compiler just tries to translate C++ into some intermediate representation as fast as possible. The linker can do the LTO thing, then split the output into multiple chunks for parallelization, and finally merge the optimized chunks back together. With LLVM, it feels like the bitcode makes this a possible compilation approach...

ygra2• 9 years ago

Doesn't link-time code generation already exist (at least on MSVC, I think). It makes the linking step so much more expensive, though, which sucks for incremental builds.

mpweiher5• 9 years ago

These numbers don't seem abnormal;

Hmm...not "abnormal" in the sense that we've gotten used to it: yes. Heck, last I heard building OneNote for Mac takes about half a day on a powerful MacPro.

But I'd say definitely abnormal in terms of how things should be.

innocenat• 9 years ago

I think that, especially in templated code, you only know which part of code can be thrown away because of the optimization.

pjmlp23• 9 years ago

Damn, are we doing this wrong?

Not using modules? Yeah I know C++'s made the mistake of not using them since the beginning and it is a long road until they are here (202x ?).

However Google was showing their modules work at CppCon 2016, so I guess Chrome does not make use of clang modules.

mysterydip• 9 years ago

Makes me wonder about a distributed compilation. If everyone had a monster workstation but not everyone compiles at the same time, a theoretical networked compiler (which may already exist as I haven't really checked) could spread the files out among available workstations and bring things back together near the end.

As the largest issue is the throwing away of duplicate work, I'd see it as a kind of reverse binary tree: machines working on files that depend on others talk together, then when finished send the condensed work up the chain (and signal their availability for the next workload chunk or phase) until everything collapses down back to the original machine.

fazzone• 9 years ago

Isn't this what IncrediBuild does on Windows?

ordu• 9 years ago

distcc: https://en.wikipedia.org/wiki/Distcc

moises_silva• 9 years ago

https://wiki.archlinux.org/index.php/Distcc

I used that 10+ years ago on Gentoo, and never saw anyone using it since. Don't know how often is used now days.

jdright2• 9 years ago

The best solution today for distributed build is: http://www.fastbuild.org/docs/home.html

It is faster than IncrediBuild, even faster than SN-DBS and has multi-platform support, the only problem is that it requires its own build script.

adrianratnapala2• 9 years ago

Ok, I see distcc helping people with computer labs and server farms, but I thought mysterydip wanted to help the Gentoo community and other such. I.e. peer-to-peer build sharing.

So what kind of cryptographic guarantees would you need for that? And if you can only verify the build results by trusting signatures from upon high, then what is the point? Perhaps those builds could be turned into work in a proof-of-work blockchain. Do compilers contain any hard-to-do, but easy to verify steps?

Whole shelves full of useless PhDs thesis are just waiting to be written on this topic.

raverbashing2• 9 years ago

C was ok. Not good, but ok.

Then we tried to attach the OO paradigm to it and we got the monstrosity that is C++ (and as a consequence of that - Java - which has fixed some issues but still suffers)

And don't get me started on templates

I'm so glad that paradigm is starting to die out and hopefully Rust, Go and others will take over (their object model still doesn't get around my head but it will eventually)

codedokode• 9 years ago

C has the same problem because it relies on includes and does not have modules.

kuschku2• 9 years ago

And luckily Java is now also trying to fix modules. Maybe, someday, those concepts will arrive over in C++ land, too.

ioquatix• 9 years ago

I built Mozilla on my PowerBook G4 and it was still going 24 hours later :D Ah, good memories.

pjmlp23• 9 years ago

I had that experience with KDE, back in SuSE 6.3.

jussij• 9 years ago

a from-scratch build of Chrome will take at least 30 minutes on a

That all.

I once work for a rather big accounting software company and the full build of their accounting product took about 4 to 5 hours to complete on the build server.

We ran the build at the close of business every day and the build engineer had to log remotely just to make sure the build worked, other wise the QA team would have nothing to test in the morning.

It too was written using the C++ language.

andrepd5• 9 years ago

Damn, are we doing this wrong?

Give me a tool that's: (i) as fast, (ii) as mature and well supported, (iii) as powerful as C++ and I will switch in a heartbeat. But until there is such an alternative it's futile to complain about the shortcomings of C++, because if you want the powerful, zero-cost abstractions, the mountains of support, and access to billions of existing lines of code, you pretty much have nowhere else to go.

lallysingh2• 9 years ago

But until there is such an alternative it's futile to complain about the shortcomings of C++

Then how does C++ improve?

p0nce4• 9 years ago

So: a C++ competitor shall beat it in every dimension you've choosen, until then it's futile to complain about the shortcomings of C++. This doesn't sounds very logical, only like a case of sampling bias.

kalekold• 9 years ago

Try Dlang.

jkantz• 9 years ago

Aspects of Unix philosophy where "your done when there's nothing left to remove" need some revival.

http://www.catb.org/esr/writings/taoup/html/ch04s02.html#com...

http://suckless.org

jdright2• 9 years ago

I understand that very well, currently the project I work has 6.2GB of object files (per platform), imagine that. Chrome is not that huge. :)

0xFFC2• 9 years ago

What project is that? I am very curious.

zx2c4• 9 years ago

TBH I don't remember toughing out a full build without resorting to goma.

I have! Every time you guys bump a snapshot, my Gentoo boxes whirl away and heat my house, compiling a new version from scratch. On an octocore Skylake Xeon laptop, this takes 2 hours 48 minutes.

copperred• 9 years ago

The important question is: how long does an incremental build take? The time to perform a full clean build is important but clean builds are rarely a part of my development flow. In my experience, efforts to reduce build time focus first on the incremental case, and rightly so.

hydromet• 9 years ago

With crap and hassle like this in the year 2017 for something as "simple" as a web browser, be sure to remind your boss Ray Kurzweil at Google, "the singularity is near" ROFLOL

deong2• 9 years ago

At the risk of looking like I take the "Singularly" at all seriously...(and putting aside the non-sequitur about Kurzweil having anything to do with Google's shipping software)...

The implication you're making is that because computers have weird little glitches that pop up to cause havoc every once in a while, then it must be laughable to imagine they could rival the marvels of human intelligence. What that tells me for certain is that you haven't paid much attention to human intelligence.

There are flat-earthers, anti-vaccine nuts, and people convinced we faked the moon landings. We'll happily argue for thousands of years over whether the kid a virgin had is or is not the same person as his dad, but not that dad, the other dad. Show me someone who intuitively understands probabilities, and I'll show you someone who incorrectly assesses how people understand probabilities. I'm pretty sure the odd bug here and there doesn't disprove the ability of machines to outthink us.

netcraft• 9 years ago

It'd be crazy to think about the energy spent over time on building just chrome - turn that into a carbon footprint and itd be shocking! I always try to write code for humans and not over-engineer or prematurely optimize code, but sometimes I wonder about a section of code over the long term - that bit of javascript thats going to be run on lots of phones and other devices over years or decades - and how much electricity its going to cost more just because I used `map` instead of a `for` loop... I can't imagine working with a build process that takes that long to run.

treehau52• 9 years ago

How did we get here? Well, C++ and its stupid O(n^2) compilation complexity [...]

Wasn't large compilation time a driving force behind coming up with Go? Is a garbage-collected language not suitable for a web browser? I am just curious because I absolutely love writing Go

wbl• 9 years ago

GC performance is a trade between throughput and pause time. There are tricks you could play with each tab in its own heap which is entirely discarded on page change, but I think it would take something expensive like Azure to really work.

jdc0589• 9 years ago

Wasn't large compilation time driving forces behind writing Go

yes, that was one of the tenants.

Is a garbage-collected language not suitable for a web browser?

In theory its fine; but there's a lot of historical baggage that comes along with garbage collection and the majority of languages that support it (e.g. almost no value types).

Golang fairs pretty well latency wise for a GC'd language. I'd be curious for someone with more experience than me to talk through a deep dive of instances where go's latency/throughput characteristics are and are not good enough for specific applications.

mmarks4• 9 years ago

Not to take away from your point, but in my experience the vast majority of C++ build pipelines even at major companies can still be improved. Few people enjoy 'improving the build', it often touches everything, and requires discipline to keep it working. Most of the projects I've worked on have been larger than Chrome, I've seen the compile time for BioShock Infinite go from 2 hours down to 15 minutes with serious work on header use, precompiled headers, and all the other tricks people use. Epic's build system is a pretty good example. There is even a older book, Large-scale C++ Design, that is specifically about this point.

Starting with a full build that initially takes hours and it shrinking to < 15-20 minutes and better seems pretty par for the course for truely large C++ projects. You don't get a fast build process for free, but if the team makes it a priority, alot can be done.

EDIT: Times mentioned were for a full build, often you rarely due a full build, incremental builds should be majority. Places that don't make incremental builds 100% reliable drive me crazy and waste so much developer time. This is common, but it's a lame excuse. Just do the work and fix it.

inglor• 9 years ago

Just as a data point - Chrome has more code than the linux kernel - Would you say BioShock Infinite seriously has a larger code base than Chrome?

I think a lot of people don't estimate correctly just how huge Chrome is.

evmar• 9 years ago

I worked on exactly this problem for Chrome! I agree with all your major points -- in particular, optimizing incremental builds is the most important thing for developer sanity.

Here's a post about what I did: http://neugierig.org/software/chromium/notes/2011/02/ninja.h...

brucedawson• 9 years ago

A lot of time has been spent on optimizing Chrome's build: - Ninja build system will perfectly parallelize the build without overloading resources (modulo this OS bug) - Meta-build system was recently completely replaced (gyp-> gn) to improve builds - Lots of work on clang-cl to allow compiling Chrome for Windows without using Microsoft's compiler - Distributed build system to further increase parallelism

So, lots of work has been done to deal with the build times. And probably not a lot of low hanging fruit to be found. But, still more work is being done. Support is being added for 'jumbo' builds (aka unity builds, where multiple translation units are #included into one) which is helping a bit with compile and link times.

threepipeproblm• 9 years ago

Wasn't this issue basically a driving force for Google creating Go? Or at least, a major design goal for Go was to get rid of O(n^2) compilation?

secure• 9 years ago

Yes, as per Rob Pike’s comments on the subject: https://commandcenter.blogspot.com/2012/06/less-is-exponenti...

phaedrus• 9 years ago

Yes, you're doing it wrong.

The trick is you've got to reduce your "saturation level" of #includes in header files, by preferring forward declarations over #includes, and using the PIMPL pattern to move your classes' implementations into isolated files, so that transitive dependencies of dependencies don't all get recursively #included in.

When it comes to templates, one has to be very aggressive in asking "Does this (sub-part) really have to be in template code, or can we factor this code out?" Any time I write my own template classes, I separate things between a base class that is not a template, and make the template class derived from it. Any computation which does not explicitly depend on the type parameter, or which can be implemented by the non-template code if the template just overrides a few protected virtual functions to carry out the details, gets moved to the non-template base class.

If your problem is not with template classes which you have written, but with templates from a library, consider that in most (all?) cases there is still some "root" location (in your code) which is binding these templates to these user-types. This root location will either itself be a (user-written) template class, or it is an ordinary class which "knows" both the template and the bound-type(s). Both of these cases can be dealt with either by separating it into non-template base and derived template, or using the PIMPL idiom, or both.

The general principle is that what you allow in your headers should be the lower bound of the information needed to specify the system. Unfortunately this takes active work and vigilance to maintain, and a C++ programmer is not going to understand the need for it until they reach the point of 30 minute builds and 1.4GB's of .o files.

deong2• 9 years ago

I've found that I generally regret doing this kind of thing to the extent that you need to do it to make a meaningful difference. The problem is that all this stuff comes at a cost -- my source code is no longer structured in a semantically meaningful way.

The SICP quote comes to mind here: "Programs must be written for people to read, and only incidentally for machines to execute." I greatly prefer to have my code organized in a sensible way. I want to know that "here is where the FooWidget code is".

It's not the end of the world, and people can adjust, but part of what I hate about working on just about anyone's Java code is this constant mental assault of "no, you need to be in the FooWidgetFactoryImpl file to find that code". Just let me have "customer.cpp" or whatever, and I'll live with grabbing coffee during the build.

Admittedly, I don't work on truly large applications. I can imagine priorities change when builds take two hours instead of the 15 minutes I might have to live with.

frankzinger3• 9 years ago

Yes, you're doing it wrong.

Your comment is great but I have spent enough time working on Chromium to know that they have people working on the build who know all of this stuff and much more. They understand the build from the top to the bottom of the toolchain stack. (@evmar used to be one of these people and he actually commented in this thread at https://news.ycombinator.com/item?id=14736611.) I am sure your parent commenter is a great developer but I get the impression he/she is not one of the Chromium build people.

struct• 9 years ago

However, they have just merged a feature called jumbo which squashes lots of compilation units together. The guy who developed it (Daniel Bratell from Opera) reckons an improvement of 70%-95% in per-file compilation time[1]. But, it's only for the core of Blink right now.

[1] https://groups.google.com/a/chromium.org/forum/#!searchin/ch...

PopsiclePete• 9 years ago

Heh. I was reading about the origins of Golang and how Rob Pike and Robert Griesemer were waiting 3 hours or something for a C++ build to finish. And with that much time available - might as well make a brand new language that has compilation speed as a first-class feature.

Are we doing it wrong? In the case of C++ - yes, absolutely, 100% certainly, wrong. I'm not suggesting that Chrome would be better off under Golang - I'm just saddened that 30 years after the C++ abomination was born in Stroustrup's head, nothing else has come out to challenge it.

Maybe Rust?

pjmlp23• 9 years ago

I prefer that C++ abomination than the unsafe by design C.

Any C++ replacement needs the love of Apple, Microsoft, IBM, HP, Google and everyone else that sells operating systems.

captainmuon4• 9 years ago

Could it be that we are doing the web browser wrong?

I think large parts of chrome actually belong into the OS. The network parts, the drawing library (skia), the crypto implementation, the window and tab management, and so on.

The javascript engine could be factored out, too, so more apps could benefit from it (without bundling a whole frickin Chromium).

Video and audio would be deferred to DirectShow, Quartz, VLC, Mplayer, ...

Ideally, what remains is just a layout engine and some glue code for the UI. It's the monolithic kernel vs microkernel debate all over again.

Plugins have a bad rep in the context of browsers, but I think this "microkernel browser" where everything is a plugin or OS library can be potentially more secure than the current state, since we can wall off the components between interfaces much better.

I also think it would be much "freer". Browsers like Firefox and Chrome are open-source, but they are free in license only. I can't realistically go ahead and make my own browser. The whole thing is so complex that you have to be Google or Apple or Microsoft to do that. The best I could achive is a reskin of WebKit. I think that would be different with a more modular browser.

naasking2• 9 years ago

I think large parts of chrome actually belong into the OS. The network parts, the drawing library (skia), the crypto implementation, the window and tab management, and so on.

The problem is cross-platform support. Depending on the OS is obvious if every OS supported the required features.

korzun• 9 years ago

You are going backward with the 'Internet Explorer' like buy-in. That approach would only be faster in one segment; everything else will be slower.

There is no point in speeding up the raw compile time by a couple of minutes if you are increasing the development and testing time by a couple of weeks.

astrobe_• 9 years ago

No, the web browser is all-righty. It has become the universal VM, so it's only natural that it is as big and slow to compile as an OS.

msla2• 9 years ago

I'd be wary of moving the crypto into the OS, because OS upgrades are few and far between. Browsers are easier to upgrade, as we know from the rather aggressive auto-upgrade cycles of Chrome and Firefox, whereas if you have bad and/or now-known-to-be-insecure crypto in the OS, well, you're stuck with it for the foreseeable future. People are still running Windows XP.

swiley4• 9 years ago

When elinks is compiled with javascript support you must provide an external javascript library.

I really like elinks, it's a shame I can't use it to view blackboard and other sites I have to use...

nialv7• 9 years ago

Well there's the NetSurf project: http://www.netsurf-browser.org

patrickmay• 9 years ago

. . . and try to create encapsulation with getter/setters.

That violates encapsulation. Getters should be rare and setters almost non-existent.

lefty2• 9 years ago

The obvious solution is to change the c++ specification so that you can modify an inline function without requiring a recompile of every file where it's included

wirelessest• 9 years ago

Heh, good to hear Google feels this pain, too. I work on the chromium codebase, and a 1 hour clean compile on a macbook pro seems optimistic.

Luckily ninja and ccache do a good job of meaning you only ever do that once (and rsync solved that problem for us). Not that a 20-second compile for a one line change is something I should be content with, but it's certainly workable.

mamcx2• 9 years ago

Damn, are we doing this wrong?

SUPER-wrong.

With all the time-loss, money-loss, opportunity-loss, C/C++/Js must have been rule out as the worst tools for our job decades ago, by ANY RATIONAL ENGINER.

And instead of band-aid them, the sane path is to freeze them and only apply critical patch and use something else. And other options are already know, have been proved and could have be a better foundation, except because developers are irrational like hell and don't wanna true progress at all.

mplewis• 9 years ago

And why should any other programming language be better?

urza• 9 years ago

Hi could you guys please fix the white flashing bug chrome has for almost 10 years? I think it might me the longest open unsolved bug I know in any IT project.

https://support.google.com/chrome/forum/AAAAP1KN0B0Rmd8IyUjG...

floatboth2• 9 years ago

30 minutes on a Macbook Pro--maybe an hour

Not bad. FreeBSD takes about the same, and it's plain C… oh wait, no, a large part of the compile time is LLVM/clang which is… not C.

Though there is now a clever incremental sort of thing called meta mode https://www.bsdcan.org/2014/schedule/attachments/267_freebsd...

rsp1984• 9 years ago

However, to actually have the C++ compiler do inlining at compile time (LTO be damned), we have to put the definitions of inline functions into header files, which greatly increases their size and processing time.

I can't think of a component in a browser that would require inlined function calls in order to be performant. To really matter it would have to be many millions of calls per second.

Moreover, because the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file!

So let me suggest that you revisit those inlined function calls again. Once you start putting them into proper .cpp files and make use of forward declarations where possible the whole header dependency graph will probably simplify quite a bit.

Don't even get me started on templates.

Right. Don't use them unless there's no other way. Especially avoid for large-ish classes that do complicated stuff. If you can spare a couple CPU cycles (and most of the time you can) determine things at run-time instead of compile time.

Of course all of this is theory, not taking into account practical matters like deadlines, code readability or developer turnover.

Full disclosure: I worked at Google, but not on the Chrome team :)

saurik• 9 years ago

There are a million responses to your comment, but unless I missed something, no one seems to be redirecting you back to the actual problem at hand and are recommending all sorts of generic ways to reduce the time required to compile a large C++ project... but the issue here was not the generic one: the entire thing that made this article interesting is that his CPU was not being used at all, so this has nothing to do with "a Chrome build is truly a computational load to be reckoned with". He even goes out of his way to show his CPU load graphs so we can see multiple seconds in a row where his computer is 98% idle and yet he still can't even move his mouse.

Your comment, and essentially every other single one on this entire thread thereby makes me wonder if anyone on this subthread read the article :/.

OK, I decided to search for the word "process", and found one person responding to you who did read the article, and depressingly only a handful of people even responding to the top-level article who apparently read the article. This entire post is such a great example of "the problem with this kind of discussion forum" :/.

https://news.ycombinator.com/item?id=14735977

REGARDLESS...

What was described in this article wasn't "Chrome's build is too slow", it was "there is a weird issue in Windows 10 (which apparently wasn't even a problem with Windows 7, and so we could easily argue is a regression) where process destruction takes a global lock on something that is seemingly shared with basic things like UI message passing". The fact that he was running a Chrome build to demonstrate how this manages to occasionally more than decimate the processing power of his computer was just a random example that this user ran into: it could have been any task doing anything that involved spawning a lot of processes, and the story would have been exactly the same.

Now, that said, if you want to redirect this to "what can the Chrome team do to mitigate this issue", and you want the answer to not be "please please lean on Microsoft to do something about this weird lock regression in Windows 10 so as to improve the process spawn parallelism for every project, not just compiling Chrome"... well, "sure", we can say you are "doing this wrong", and it is arguably even a "trivial fix"!

Right now, the C++ compiler pipeline tends to spawn at least one (if not more than one) process per translation unit. If gcc or clang (I'm not sure which one would be easier for this; I'm going to be honest and say "probably clang" even though it feels like a punch in the gut) were to be refactored into a build server and then the build manager (make or cmake or ninja or whatever it is Google decided to write this week) connected to a pool of those to queue builds, you would work around this issue and apparently get a noticeable speed up in Chrome compiles on Windows 10, due to the existence of this process destruction lock.

One could even imagine ninja just building clang into itself and then running clang directly in-process on a large number of threads (rather than a large number of processes), and so there would only be a single giant process that did the entire build, end-to-end. That would probably bring a bunch of other performance advantages to bear as well, and is probably a weekend-long project to get to a proof-of-concept stage for an intern, come to think of it... you should get on it! ;P

Dylan168074• 9 years ago

Your comment, and essentially every other single one on this entire thread thereby makes me wonder if anyone on this subthread read the article :/.

Just let people talk about what they want to talk about, maybe? The main problem in the article is interesting but far less actionable than the overall situation of slow compilation.

Do you want things like rampant speculation and insulting windows 10? Do you expect everyone to pull out kernel debuggers to be able to make directly relevant comments? It's okay to talk about a related issue. Concluding that they didn't read the article is kind of insulting.

titzer2• 9 years ago

You're right. I didn't read the whole article. I skimmed it. But I did actually pick up on the locking problem and see the graphs with huge amounts of idle time. I knew this comment was only marginally related to the actual article. So, :-/

However I suspect that as soon as that lock regression in Windows is fixed, that monster CPU load is coming back home to roost, and the workstation is going to be just as dead as my 64 core Linux workstation has been when I've actually '-j 500' without gomacc up and running correctly.

So, by all means, Microsoft should fix this lock regression.

But there's this...elephant...in...this room here.

josteink• 9 years ago

A Chrome build is truly a computational load to be reckoned with. Without the distributed build, a from-scratch build of Chrome will take at least 30 minutes on a Macbook Pro--maybe an hour

From someone regularly working on multi hour builds (on build servers), this imo sounds like the light kind of build I'd wish I worked with ;)

That said, this article was about a weird behaviour in Windows when destroying processes, not about heavy builds.

elorant• 9 years ago

We've come to the point where building the browser from scratch takes more than building the OS itself.

laumars• 9 years ago

That's because we've come to the point where the browser is now more akin to an OS than a networked document reader.

bittermang• 9 years ago

I'm reading this thread on a Chromebook and really getting a kick...

oblio2• 9 years ago

That's because the browser had to do things the sneaky way.

Everybody wants to control the app platform. Every time a cross platform solutions appears everyone tries to shut it down. See: Java, Flash, etc.

The only one that platform owners couldn't really shut down (though they are getting better at it), is the web. It was considered too "dumb" to shut down and too useful to completely avoid. And bit by bit, just like the boiling frog story, people added interactivity features until we arrived where we are now. A bloated, perhaps crippled cross platform solution, but the best we have now for running things on things from embedded, mobile, desktop, etc., etc.

qb454• 9 years ago

Somebody said Chrome takes 30 minutes to build on one workstation and I don't seriously think you would build Windows 10 that fast.

Beltiras• 9 years ago

Interesting metric actually. Building Windows takes 12 hours [1]. It's a bit harder to find metrics on Linux. You can build the kernel in 60 seconds apparently [2] but that is not a complete operating system.

[1] https://stackoverflow.com/questions/226377/operating-system-... [2] http://www.phoronix.com/scan.php?page=news_item&px=MTAyNjU

shortsightedsid• 9 years ago

Someone's old data from 2012 to build a core-image-sato using Yocto is just over an hour - http://www.burtonini.com/blog/2012/11/15/yocto-build-times/.

That said, core-image-sato used to be just a simple demo image. In my old build system, we would build the Arago Project for a TI SoC every night and it would take a few hours but that included a lot of the DSP code as well which was really slow too. So a couple of hours on average is my guess.

cat1992• 9 years ago

Can build base+X11 BSD systems in a handful of minutes on modern hardware in one command. I don't have modern hardware, and build over NFS, so I can't give a precise figure.

But yes, windows includes lots of API's and frameworks and GUI apps and such. Probably something comparable complexity wise would be Base+X11+(1 of KDE or Gnome)..

rkeene2• 9 years ago

Building my Linux distribution from scratch takes about an hour on a modern system from top-level ./configure && make -j5 (which produces the installer ISO image), a great deal of the time, however, is spent in first building the cross-compiler toolchain (to ensure that no matter where you build my Linux distribution you get the same results).

This is the cross-compiler toolchain, the kernel, and about 250 external packages.

int_19h2• 9 years ago

How do you define the OS? If it's just the kernel, sure, but then is it really surprising? If it's also all the userspace necessary to e.g. run said browser, then I don't think this will be true anymore.

rbanffy7• 9 years ago

I am pretty sure building a common set of packages for, say, Fedora, or any version of Windows takes a couple orders of magnitude more resources than building Chrome.

On my Macs I've been doing "port install -s" most of the time and I can tell you some relatively small installs can take a very long time.

kbart• 9 years ago

"We've come to the point where building the browser from scratch takes more than building the OS itself."

It doesn't. It took ~5h to build a tiny (~150 MB) complete system (Linux kernel + Yocto based "OS") from sources on 4 core PC few years ago. With modern CPU, I guess, it might be 3h or so. On the side note, the build process generated >50GB of files.

krylon4• 9 years ago

I admit I have not done so in a while, but I vaguely remember building Net/OpenBSD from source was faster than that on moederately powerful hardware (Core2 Quad 2.4 GHz, 8GB RAM, no SSD).

I think NetBSD took ~2.5 hours, including building its toolchain.

restalis• 9 years ago

If "OS" isn't restricted to *nix then we have ReactOS that builds in around 10 minutes!

gorkonsine• 9 years ago

Honestly, Yocto is a complete mess and a terrible embedded OS system IMO. Buildroot is far better, and takes far less time to build.

nomercy400• 9 years ago

Does this also occur on other OSes, like Linux, or MacOS? Can you move your Chrome build to another OS and not experience this problem?

hoodoof• 9 years ago

Saw the headline and thought "must be Windows".

I'm not a Windows hater, but one of my long standing gripes about Windows is that it just seems to have terrible multitasking compared to OSX.

I'm sure there are reasons but it just seems utterly symbolic of Microsoft that they never managed to get Windows to multitask in a rock solid, smooth and reliable way like OSX.

gambiting• 9 years ago

Uhm I have a 40-core machine with 256GB of ram at work and I can make it completely unresponsive without taxing either ram or the cpu to 100% - we have a certain computational load that just destroys the CPU<->Ram bandwidth, so cpu usage is about 30-40%, ram sits at 50%, and yet the computer is completely unresponsive. It's exactly the same on both Windows Server and Linux - we're just running into hardware limits.

chemmail• 9 years ago

Interesting. I've been CPU mining on my old IVY bridge and i don't even notice it causing much performance drop. Free coins for me!

hota_mazi• 9 years ago

It seems to have improved these past years but beach balls used to be comically widespread on OS X. I'm not at all convinced OS X is in any better shape than Windows.

And OS X certainly doesn't have anywhere near all the kick ass monitoring tools that Windows as, such as the ones shown in this article.

hultner• 9 years ago

You do know that OS X have DTrace?

rbanffy7• 9 years ago

And OS X certainly doesn't have anywhere near all the kick ass monitoring tools that Windows as, such as the ones shown in this article

It certainly helps that Intel really likes Windows, but I haven't seen Mac users complaining as much as Windows users do, so, it may be for lack of demand (and the bundling of dtrace, for ages).

walterstucco• 9 years ago

OS X just slows everything down so you won't notice when something get stuck.

I've been a Mac user for 7 years now, my latest greatest MacBook Pro halts for a few seconds when I connect an external monitor and everything is frozen and unresponsive, like I'm looking at a screenshot of the system

OS multitasking is still usually shit...

adrianN• 9 years ago

My ten year old crappy MacBook Pro doesn't do that. Weird.

qb454• 9 years ago

That's at least excusable, unlike GUI freeze caused by termination of a number of processes which don't even use the GUI.

href• 9 years ago

I'm reading your comment while upgrading brew. This prevents me from working because typing in Sublime Text slows down to a crawl during brew's compilations ;)

gruturo• 9 years ago

I am absolutely a Windows hater, but I've experienced the same problem in Linux, MacOS, Windows, BSD, you name it.

See for example Con Kolivas' famous rant about how Linux schedulers were ignoring the interactivity requirements of desktop usage, and resulted in a terrible experience with constant tiny freezes.

rbanffy7• 9 years ago

my long standing gripes about Windows is that it just seems to have terrible multitasking compared to OSX.

Using both Macs and Linux laptops, I'm sometimes shocked at how the Mac sometimes locks up when the Linux machines degrade much more gracefully under heavy loads. I never dug too deep into it, but it feels like it's something with HFS+ under heavy IO. I hope APFS fixes that.

mort962• 9 years ago

I'm far from an expert on the topic, but I'm pretty sure APFS does a lot of global locking, which would definitely explain at least some of that.

hsivonen• 9 years ago

Interesting. Back when I switched my Firefox builds from Core2 Duo T8300 @ 2.40 GHz + spinning disk + OS X to i7-950 @ 3.07GHz + SSD + Ubuntu, GUI responsiveness during Firefox build went from OK to bad. Canonical's support suggested getting a second computer as a build machine.

eddieh• 9 years ago

Ever since I purged Dropbox from my life I haven't had intermittent UI lockups on macOS.

ksk3• 9 years ago

This has little to nothing to do with multitasking.

nailer• 9 years ago

While the issue closing processes slowly is unique to Windows 10, I've found similar situations of not being able to control my OS on RHEL, Fedora, Ubuntu and MacOS.

At this point I genuinely believe latency will be the death of general purpose computing.

iOS and Android very rarely get out of control. Apple is already pushing iOS devices as laptop replacements. But we lose a lot on these devices with their locked down OSs and inability to install the software we want.

m_fayer• 9 years ago

All the midrange Android devices I own get out of control all the time if you switch between heavy apps too quickly. And every once in a while while I wait for the system to become responsive again, some low level process like "Android System" will crash and restart itself. I shudder to think what will happen once we start throwing desktop-class loads at Android using it's current application stack.

SmellyGeekBoy2• 9 years ago

I've seen Android get "out of control" on lower end devices. If you stick with higher end devices and/or first-party the experience is usually very slick. I don't think I've ever had a OnePlus device freeze up or stutter on me, for instance.

Dayshine• 9 years ago

My OnePlus X certainly does. When charging or after 10/15 minutes of intense graphical use it starts getting sluggish and unresponsive.

peoplewindow• 9 years ago

Android isn't locked down. But it also isn't exactly smooth. my Android device frequently shows me white screens or startup splashes when doing something as simple as switching tasks. It clearly can't keep multiple apps in memory at once. Somehow iOS manages to produce a much smoother experience. I don't think it's a fundamental issue though. Just a matter of priorities.

Androider• 9 years ago

Android 7 on a Pixel device, and just browsing with Chrome will cause it to completely freeze (no panning, can't focus the URL, can't open task switcher) for seconds when first landing on a heavy page. This has been a problem since day one and with every single Android device I've ever used, as there's still a whole bunch of stuff on the main UI thread as any developer on Android will intimately know, and it absolutely will freeze when under a load. It's just fundamentally architected wrong, since you can block the UI thread it does block.

paulio2• 9 years ago

He's also written some stuff regarding Visual Studio perf.

https://randomascii.wordpress.com/2014/04/15/self-inflicted-...

Close to my heart as I use Visual Studio all day. Horrid piece of software.

I'll probably get downvoted for this.

qb454• 9 years ago

I'll probably get downvoted for this.

Obliged :)

No, seriously, why are people doing that?

bitwize2• 9 years ago

It's against HN policy to downvote-bait.

SmellyGeekBoy2• 9 years ago

I live in two worlds at the moment - supporting and maintaining a large PHP application (for which I use phpStorm) and developing .NET applications (for which I, obviously, use VS).

I've been a VS user for nigh-on 10 years now and always found the experience really fantastic. I think it's certainly more coherent that the IntelliJ-based IDEs (which, to be fair, are also very good).

Anyway, I'm genuinely surprised to see VS described as "horrid" - what specifically do you dislike about it?

ygra2• 9 years ago

VS for .NET is a different beast than VS for C++, the latter being a far less tool-friendly environment. There's also the point that newer versions sometimes improve a lot over older ones, but licensing costs may keep people on older versions for too long (that, and the fact that with large C++ projects an upgrade is rarely straight-forward). This then ends up with people hating VS when all they've used is VS2008 on C++.

Not that I don't have any complaints (although those may very well be due to certain extensions), but I've always found VS to be much more responsive, stable, and featureful on the .NET side than C++; much of that certainly due to complexities of the respective languages.

paulio2• 9 years ago

"what specifically do you dislike about it?"

It's not the tooling I dislike it's the performance. No matter what I throw at it I still get the same experience. It feels like 95% of everything it does is blocking the UI.

43224gg252• 9 years ago

FWIW I agree with you. Visual Studio looks like it was designed by fisher price, the whole thing takes up ~60gb of space, and if you're not running it on a high-end work station it's likely going to lag when editing large code bases. Not to mention it's closed source software.

bitwize2• 9 years ago

The Amiga prioritized user-input interrupts before all other interrupts, so if there ever was a time you couldn't move the mouse on the Amiga, it meant that the system was well and truly crashed.

30 years on and the peecee industry still doesn't know how to design a fucking system.

amiga-workbench• 9 years ago

I was just thinking that, my A1200 never skipped a beat.

ComodoHacker• 9 years ago

I remember every other book on Windows programming saying "Process creation/destruction is expensive, use thread pools (or at least process pools) instead, that's the way to go on Windows". Perhaps this mindset is ingrained for Windows QA team too - they don't have [enough] test cases for such scenarios.

usefulcat2• 9 years ago

Seems like a feedback loop. It's expensive, so most apps avoid doing that, meaning there's less need to check for performance regressions. Then if there is a regression, it further increases the incentive for developers to avoid it, so in the future even fewer apps will do that, making it even more of an unusual use case, so even less need to test for it..

koffiezet• 9 years ago

This is probably also why Cygwin and even the WSL subsystem in general are a lot slower when running more complex shell scripts, which is typically spawns tons of processes.

I wrote a pretty simple shell script to test WSL process spawn speed, which loops over a simple echo piped to a grep, and add 1 to a counter until it reaches 1000.

On my windows machine, in a Linux VM, I consistently get times like this:

  real    0m1.381s
  user    0m0.073s
  sys     0m1.472s

On the same machine in WSL, I get results like this consistently:

  real    0m14.878s
  user    0m0.469s
  sys     0m12.109s

That is 10 times slower... I don't have cygwin installed anymore, but when I tested it initially when trying out WSL, it was even slower...

stargrazer• 9 years ago

I think the best summary of how to do this is to take a look at Herb Sutter's three article set on "Minimizing Compile-Time Dependencies". https://herbsutter.com/gotw/

"The Compilation Firewall", or pimpl idiom, is a clever mechanism for getting code out of the header: https://herbsutter.com/gotw/_100/

cheez3• 9 years ago

I use this liberally. Generally, it is a runtime performance issue only on repeated allocations for which you can optimize if needed. Once-only allocations can be ignored when using compiler firewalls.

lallysingh2• 9 years ago

And the second cache miss: 1 for the cache miss, 1 for the actual object.

hl5• 9 years ago

I wonder if the "privacy" features in Win10 play a role here. Seems like some extra process accounting could cause delays not present in previous versions.

dankohn1• 9 years ago

I feel like the (brilliant post) is missing the context that if you were debugging the latency on Linux, you would have the source code to continue the investigation until you found and fixed the problem, as opposed to just teeing it up for Microsoft.

hghwng• 9 years ago

Most of the problems are solvable, if proper tools are used. Microsoft provides "PDB" files, which contains symbols for ease of debugging. You can get them from Microsoft's symbol server. Load the symbol and the binary in IDA, and the generated pseudo code is enough for most scenarios.

In theory, debugging programs on Linux should be easier. However, for some distributions (like Arch Linux) debug symbols are not provided. You have to compile the program on your own if you want to debug. It's especially painful if the target program has a large codebase!

martyvis• 9 years ago

For my own amusement, when ever I get a new OS build on my machine, I'd open up a task manager and watch CPU load just by wiggling the mouse a lot or maybe simply pressing page up and down. I'm pretty sure it's always pretty easy to generate 25% CPU doing very little. Another thing would be just opening a local file from within a running application and wondering why multiple seconds and hence billions of CPU cycles seems to be consumed what one expects should be doing a fairly menial task. (I am pretty sure in DOS 3.3 with Norton Commander it was quicker )

stinos2• 9 years ago

I'm pretty sure it's always pretty easy to generate 25% CPU doing very little

not for me, just tried mouse wiggling, scrolling, page up/down in browsers/file managers on Windows 7, 10 and Ubuntu 16 and none of those have such behaviour. Likewise for opening a file dialog. And I didn't expect otherwise actually, what OS are you talking about?

ptero• 9 years ago

I wonder if this accurately captures the time spent on the task. I am surprised opening a file takes that long. I primarily live in a Linux environment, but I also run some Java code for R&D data collection on Windows also. It opens 10-20 files on startup which does not lead to a perceptible delay. I have not measured exact performance, but if opening (all 20) took more than half a second or so I would have surely noticed this.

zlynx2• 9 years ago

That used to be true back when the CPU did all of the GUI rendering. But now most all of it is offloaded to the GPU. Any GPU that can render Quake 3 Arena at 120 FPS (and that's ALL of them even Intel IGPs) can wiggle a window around very easily.

Not sure about file opens. Simple applications like GVIM don't seem to have seconds of delay for me, but I know what you mean with things like spreadsheet or word processor files. I guess it is all of the unzipping and XML processing.

Narishma• 9 years ago

He said wiggling the mouse, not a window.

StillBored3• 9 years ago

Actually, its not as accelerated as it was back in the winXP days. Here is a random link about it: https://www.youtube.com/watch?v=ay-gqx18UTM.

Basically, the GPU card vendors could hook any part of the win32 GDI pre windows vista, and they did. Post vista, only a tiny portion of the GDI is accelerated. In theory you can avoid this by writing your application using a more modern API, but the vast majority of native windows applications continue to basically be GDI based due to age, or various GUI toolkits still being GDI based. Worse there are a number of toolkits (or browsers) which implement their own drawing routines rather than calling the system supplied ones.

The final composited results with aero are of course accelerated, but that only really tends to add additional latency. Switching to one of the basic themes makes win7 noticeably more responsive, but also tends to tear a lot. I've got an incredibly high end desktop machine, carefully tweaked/optimized and I can frequently see it the ~1/10 of a second lags while windows update after being maximized/etc. Compared to the 10 year old pretty high end XP machine (with upgraded SSD/etc, and also carefully tuned) it doesn't seem to be faster on basic desktop type operations. Fire up a recent game, or doing builds its massively faster but running word/firefox/whatever the old machine "feels" faster.

(tuned, as in I have a dozen or so, tweaks I've been collecting/researching for the past decade+, on ways to make the machine feel more responsive, it all started with MenuShow delay in win95, and has grown from there and now includes all the usual stuff plus tweaking power profiles, and a bunch of less obvious "feel" things like high DPI mice with fast base speeds).

astrodust2• 9 years ago

macOS has always had really smooth window wiggling because each window is a separate layer rendered on the GPU. Windows has had problems in the past because wiggling windows causes repaints in the background ones, they had no such buffer, but Windows Aero uses a model a lot like the macOS one.

What OS are you running?

ericfrederich• 9 years ago

I had the same exact problem. A machine that is overkill spec wise but the mouse and keyboard would freeze up every minute on the minute.

I tracked it down to my desktop wallpaper being on a rotation. Seriously... how bad does that have to be implemented in Windows to actually hang the mouse and keyboard?

Eridrus3• 9 years ago

For all the hate Windows gets, UI lockup hasn't really been a pervasive issue for me in years, whereas on my Linux desktop that's just a typical day.

_arvin3• 9 years ago

My Linux desktop never locks up. I'd suggest messing around with a different kernel, double checking graphic drivers, etc.

faragon2• 9 years ago

Why the OS don't prioritize UI threads on one or two cores?

mac01021• 9 years ago

How does it know which threads are UI threads?

vivekseth• 9 years ago

I think it should at least know which thread handles mouse movement.

usefulcat2• 9 years ago

Won't help if the problem is as the author describes (contention for a system-wide lock).

asveikau• 9 years ago

I think the windows scheduler might already do this, but the issue is that UI message queues take a highly contended lock.

snarfy2• 9 years ago

My 16mhz 68000 based Amiga 500 had smoother mouse movement than my 3.2ghz 8 core desktop.

rbanffy7• 9 years ago

IIRC, on the Amigas the CPU was barely involved in the process of reading the mouse events and moving the cursor sprite on the screen.

snarfy2• 9 years ago

Yep, hardware sprites.

onetokeoverthe• 9 years ago

There's too many spinning wheels. Stop it! People are going to just stop using the internet. If I could, I would. But I can't. So please just build simple websites!

coldcode• 9 years ago

Try running Carthage update for an iOS app with 80 frameworks. Ever seen a Mac hit 55GB of ram? MacOS kills it. It takes many of these runs, and all this does is fetch prebuilt binaries. Yes, everything I said here is beyond stupid.

sixstringtheory• 9 years ago

Is it necessary to update _all_ of them at once? I think that's the same problem as OP... rebuilding the whole world is slow, so why not do it incrementally?

Also, 80 frameworks ⊙⊙ I'm assuming some of these are internal, and written in Swift, meaning they can't be compiled into static libs (easily).

cat1992• 9 years ago

Can take the alternate approach -

Run your builds in a hugely underpowered VM, and wait much longer.. your regular usage will be largely unimpacted, although the builds take longer.

Source: Currently running a ~1000 package dpb(1)[1] build of my needed openbsd ports on a dual-core KVM machine hosted on a 8-9 year old amd64x2 2.2ghz. 3 Days and counting, will probably be done around next weekend.

From there, incremental updates are mostly slight, and can complete overnight from a cron job.

.. [1] https://man.openbsd.org/dpb

metalliqaz• 9 years ago

Out of curiosity why do you keep such an old machine around? I have found that newer machines can be had for free, and 7-year-old Xeon servers can be had for <$200 from IT recyclers.

gwern• 9 years ago

The electricity consumption numbers are also relevant.

wjd2030• 9 years ago

I just wanted to say good work. I'm impressed you dug so deep. A fix here could really impact the entire Windows 10 user base.

fooker• 9 years ago

Use the gold linker if your setup permits it.

This issue went away for me when I switched.

iainmerrick4• 9 years ago

The first time I had to compile a Linux kernel for Android, it took only a few seconds (on the ridiculously overpowered "build machine" my employer supplied). I was sure I must have done something wrong, but no, that was the entire build. It takes longer for Android to reboot than it does to build the kernel.

It does feel like there's something seriously wrong with the massive C++ codebases we use these days for key infrastructure like browsers, and the massive compilation times we put up with.

ksk3• 9 years ago

Speaking of W10, there was another annoying W10 bug where if you started typing immediately after using the touchpad there was a random delay. If you care about latency and responsiveness, it makes you want to scream at the people who implement these features.

mrstone• 9 years ago

That's not a bug. It's some kind of feature by Symantec that can be disabled in the settings.

ksk3• 9 years ago

Haha, it never crossed my mind that someone would implement this purposely. I don't have a W10 laptop so I've only noticed this on other peoples machines.

BurningFrog• 9 years ago

the C++ compiler needs to see full class definitions to, e.g., know the size of an object and its inheritance relationships, we have to put the main meat of every class definition into a header file!

Without knowing anything about modern C++ and its compilers, this seems fixable.

I'm thinking a header compiler "hint" indicating the size of an object. When compiling the full class, you get an error if the number is wrong.

goda90• 9 years ago

My workstation was recently "upgraded" from a Window 8.1 workstation with a 4th gen i5 to a Windows 10 laptop with 5th gen i7. The extra RAM and SSD over HDD is great, but whether it's because it went from a quad-core to a dual-core hyperthreaded CPU, or because of the jump to Windows 10, the mouse lag is considerably more noticeable now. I've convinced them to upgrade my laptop again, but now this article doesn't give me much hope for my new work toy.

quotemstr3• 9 years ago

Why is everyone talking about how C++ compilation is slow or something instead of talking about the real problem, which is that GDI is doing resource cleanup on process exit, under lock, for console-mode processes that have probably never used any GDI resources?

yuhong5• 9 years ago

mxatone mentions that "Win32k locking design is just bad" in https://twitter.com/mxatone/status/884436870955913216

quotemstr3• 9 years ago

It's not even bad. It's neglecting any kind of "has this process ever touched win32k.sys" test.

bane• 9 years ago

Seems like a good way to deal with this (outside of trying to convince Microsoft to fix it) is to spin up a VM and just give it a few cores, and do the build inside the VM to isolate this behavior.

arwhatever• 9 years ago

Wasn't this the title of a Bruce Springsteen song?

drumttocs8• 9 years ago

When I converted from Windows 7 to 10 I noticed that I started getting audio latency/glitches/squelches from my external audio interface- a Focusrite Scarlett 2i4. The mouse would also sometimes hang. Doing some basic tests, the problem seemed to be coming from network drivers... but I could never resolve it. I wonder if the author's discovery has anything to do with the issues.

hindsightbias• 9 years ago

What Knuth said: "Premature optimization is the root of all evil"

What most developers hear: "Optimization is the root of all evil"

rebootthesystem• 9 years ago

Thank you for bringing attention to this. Experiencing this on our W10 workstations.

I hope MS does something about this immediately. It's maddening.

lobo_tuerto• 9 years ago

I have a new Ryzen 7 CPU (you know, 8 cores, 16 threads), with 64GB RAM and for HDD a Samsung 960 PRO M.2 drive.

But when I went and plugged in an external Seagate 4TB drive, and tried to "dd zero" the s#it out of it, my whole system became unresponsive after a while, obviously I had to reset the machine as it wouldn't "kill -9" the process that made the system unresponsive.

Trying to type was a no go as keys would sometimes become "stuck". Moving the mouse around was an exercise in predictability too.

All this happened in the latest Ubuntu 17.04 64bit... #sadstory

24-core CPU and I can’t move my mouse

Related Stories