You Got Your Web Browser in my Compiler

http://randomascii.wordpress.com/2014/03/31/you-got-your-web-browser-in-my-compiler/

by joosters • 12 years ago

275 71 12 years ago

randomascii.wordpress.com

octo_t• 12 years ago

Awesome, in depth writeup (especially for someone who isn't at all familiar with Visual Studio / development on Windows in general).

From your report this is 'use a builtin XML parser' and that xml parser has a dependency on IE?

jjaredsimpson• 12 years ago

shoutouts to xperf. it is truly an amazing tool.

Flow2• 12 years ago

Years ago, I've used procmon.exe and saw VB6 load project files by doing tons of 1 byte reads...

I wonder if the behavior in the article also applies to C# compiles?

josteink• 12 years ago

MSBuild has come a long way since the VB6 days.

C# compilation is seriously fast compared to C++ and the likes (due to its proper staticly typed nature) and this allows the compiler to assert certain things quickly and not waste any more time on this.

Given how .NET developers loves separating things into tons of DLLs and libraries (and how .NET makes this easy), I would be surprised if this part of the compilation wasn't as optimized as the rest of the process.

jacobparker• 12 years ago

due to its proper staticaly typed nature

I'm not sure what you mean by this - C++ compilation isn't slow because of a lack of static typing :)

Here is some actual info on the subject: http://www.drdobbs.com/cpp/c-compilation-speed/228701711

bhouston4• 12 years ago

C# compilation is fast in part because of the removal of headers and the replacement of them by well defined assemblies with interfaces. C# compilation is fast for the same reason Java compilation is fast. :)

C++ compilation is sort of screwy in that to compile a new C++ file the compiler actually has re-compile all the definitions of the libraries you are using. It is insanely inefficient compared to just using explicitly defined and immutable API interfaces on Assemblies/Modules.

Locke16895• 12 years ago

C# doesn't have a static analyzer :)

Right now the closest thing is FxCop or a third-party tool.

bhouston4• 12 years ago

You need to reduce the parallelism in your build. If it is possible in your build setup to be runing 144 concurrent builds you are trashing your caches. This isn't an optimization. You need to reduce the number of projects being built concurrently (or even the number of total projects.) While I advocate a bit of over allocation, you have it in the extreme and this is really not optimal at all.

Also you need to get using the precompiled headers if you can -- sometimes this requires structuring your projects right to avoid lots of little projects.

Even with large projects I usually have really low compilation times for C++.

Here is a blog post I wrote about the right way to do this:

http://exocortex.com/blog/speeding_up_cpp_builds_on_windows

ygra• 12 years ago

I think Bruce already knows plenty about making builds fast ;-)

The linked article demonstrates some ignorance as well by noting that command-line MSBuild builds are slower than from Visual Studio. I'd hazard a guess and say that they didn't find the /m option.

taspeotis2• 12 years ago

To my knowledge VS caches the MSBuild projects (.targets etc) while it's running. So it's possible that, even if VS and MSBuild have the same degree of parallelism, not having to parse the MSBuild XML gives VS an edge.

bhouston4• 12 years ago

I wrote the linked article, so I take responsibility for any ignorance. It was some time ago (three years) so it maybe that we were not using the /m option at the time with MSBuild.

nknighthb3• 12 years ago

A standard test for scheduler responsiveness in Linux is to fire up a kernel compile with unlimited jobs[0]. If responsiveness is lost, the scheduler isn't ready for prime time. You think Microsoft wants the answer to this issue to be "we can't even match Linux"?

[0] Well, it used to be. I assume they've moved on to more demanding tests now, since hardware has far outstripped kernel build complexity.

bhouston4• 12 years ago

The issue isn't responsiveness during build for me (that isn't usually an issue, my build machine is generally responsive), it is compile performance from start to finish. If you over-allocate to the extreme you will trash your CPU caches and this will slow everything down. You want to minimize CPU context switches (thread switches) while also ensuring that your CPUs are running at nearly 100% -- it is a difficult balance, but unlimited jobs and focusing on the responsiveness of another task is not the solution to maximizing start-to-finish compile time for a large project.

AaronFriel2• 12 years ago

Is this the same thing as with Visual Studio compiling 12 independent projects using 12 threads each? Does the kernel compile with unlimited jobs keep the number of concurrently running jobs below the processor limit, or are additional jobs in any way blocked on cheap primitives, such as waiting for previous jobs with a mutex?

The issue here alone is that he was trying to compile 12 projects simultaneously each with every logical processor he has. There might be some slack in the system, but is each project 92% wait time?

asveikau• 12 years ago

Sounds a bit like apples and oranges. If in your test GCC happened to stall because of a bug in some shared library in user mode, would you blame the scheduler?

Maybe this isn't exactly a great argument since I know Linus has been known to take the side of not ever breaking user mode wrt kernel changes (probably rightfully so) but it is possible to write buggy code, including hangs, that runs on Linux. That would be the analogous scenario.

amaks2• 12 years ago

I think you're missing the point. The problem described in the article is the expected behavior of the indirectly loaded component (mshtml.dll) which creates message loop (COM STA requirement), and because compiler runs as a low priority process, this message loop causes priority inversion which affects entire system. There are many wrong things here: 1. COM uses message loop and hidden windows to serialize calls. This is pretty stupid IMO, but I'm pretty sure Raymond Chandler would find millions of good reasons why it's done so. 2. Layering problem (XML parser uses mshtml.dll etc etc). 3. Why use config in XML file? Isn't XML so 1990s?

Locke16895• 12 years ago

I think you mean Raymond Chen. ;)

brucedawson• 12 years ago

You need to reduce the parallelism in your build.

I looked at your blog post and it recommends using /MP and setting num. parallel project builds to num. cores.

In other words, you recommend exactly what I was doing!

We do use precompiled header files. And we get great build times. It's only with /analyze builds that the system locks up.

Yes, 144 parallel compiles is excessive, but as your article says, 12-way parallel compile and 12-way parallel project builds are both necessary to expose all possible parallelism. Otherwise there will be times when many CPUs are idle. Until VS gets a global scheduler to avoid over subscribing the CPUs there is no way to get full parallelism without sometimes over subscribing.

I'm surprised that you find SSDs that helpful. With enough RAM they should offer only modest improvements to compilation, because source files and header files get cached.

bazzargh• 12 years ago

This looks like the same bug, back in 2010: http://social.msdn.microsoft.com/Forums/fr-FR/155b51f7-60ea-...

(ie that MSXML is calling URLMon, with unpredictable results). There's a comment at the bottom of that thread that suggests how to fix the bug if it's in your own code, but not if it's in Visual Studio...

edited to add: related bugs crop up in many programs that use xml parsers, not just MS. Some combination of disabling validation, loading of external entities, or adding an entity resolver is usually needed (see eg these options for Xerces http://xerces.apache.org/xerces-j/features.html). The symptom there was usually errors in production that didn't occur on the developer's box because in production the network connection to grab the schema didn't work...or worse, a SOAP service that will fetch external entities in messages.

acqq6• 12 years ago

It looks it's urlmon.dll there, whereas Bruce observes mshtml.dll loaded.

yuhong3• 12 years ago

It was urlmon that was loading mshtml.

yuhong3• 12 years ago

In this case, I can tell from the call stack that it is calling http://msdn.microsoft.com/en-us/library/ms762722%28v=vs.85%2... with a res:// URL.

One solution would be to load the resource manually using Win32 APIs, convert to BSTR, and then calling http://msdn.microsoft.com/en-us/library/ms754585%28v=vs.85%2... instead. And yes don’t forget to disable loading of external entities too.

prayerslayer• 12 years ago

I wouldn't say that VS loads "Internet Explorer", that's a bit of an exaggeration. iexplore.exe is Internet Explorer, mshtml.dll holds the DOM implementation it's using [1,2]. Now that's still close to a web browser, but one may also use Trident to work e.g. with generic XML documents. I don't know if VS does that and whether this is a good idea or not though.

[1] N. Zakas: High Performance Javascript, p. 35 [2] http://en.wikipedia.org/wiki/Trident_(layout_engine)

pjc50• 12 years ago

He does claim that it creates a window, even if that's a conceptual one rather than a physical one. Hence the lock contention in the desktop window manager.

danielweber• 12 years ago

The question “why is my computer unresponsive when doing dozens of simultaneous compiles” is difficult to answer

Doctor, doctor, it hurts when I do this!

fafner• 12 years ago

Ah, the wonders of XML. Somehow somebody decided to use XML to store some text data. Because after all you already have a parser for this in the approved tools. So why bother using something different?! And look it does all the cool stuff like namespaces and validation and amazingly it can even fetch remote DTDs by loading IE...

simias• 12 years ago

I hate XML as much as the next guy but in this case I would blame a poorly designed (or maybe very misused) API.

There's no reason why any file parsing library would end up fetching remote data without being explicitly asked to do so. Actually, it shouldn't even be the library's concern to fetch those files, an XML library has no business with networking. It's a security concern and a maintenance hell.

acqq6• 12 years ago

There's no reason why any file parsing library would end up fetching remote data

It's not the API, it is the part of many of the specs, as it was thought to be a good idea once, specifically many standardizations involved additional definitions located at the http servers, example:

http://msdn.microsoft.com/en-us/library/aa468557.aspx

Which was "poised to play a central role in the future of XML processing, especially in Web services where it serves as one of the fundamental pillars that higher levels of abstraction are built upon."

So you had to implement it to be "conforming," and then to avoid overheads as "optimizations." Ironically, "/optimize" feature isn't optimized.

The reason this it's not discovered earlier is that the Visual Studios which contain that option were priced more thousands of dollars (I don't know the what the currently cheapest version containing "/optimize" is -- anybody knows?).

pwg2• 12 years ago

There's no reason why any file parsing library would end up fetching remote data without being explicitly asked to do so.

I would agree. But the XML spec. authors clearly disagreed with both of us:

External Entities:

XML 1.0: http://www.w3.org/TR/2008/REC-xml-20081126/#sec-external-ent

XML 1.1: http://www.w3.org/TR/2006/REC-xml11-20060816/#sec-external-e...

This is "XML speak" for an "include" statement to something like cpp, with the exception that this "include" could end up performing remote network fetches to acquire that which is being included.

So, technically, to be a proper, standards compliant XML parser, the parser has to at least submit requests to "fetch" these entities to the higher level code using the library, and let that code decide what to do about the "includes".

As to why Microsoft's implementation is the way it is, absent a Raymond Chen blog post explaining the why, we can only guess.

psuter• 12 years ago

I am not sure whether it is still the case, but the Scala compiler used to have a dependency on Swing. The reason was that one of the potent -Y flags gave you access to an AST browser after the phase of your choice. Very convenient tool for compiler plugin developers, but a questionable dependency for most.

(Edit: seems to have been taken away at least as of 2.10)

jfim• 12 years ago

scalac -Ybrowse:typer Foo.scala still works on 2.10.3. It's also pretty convenient when writing macros, not only for compiler plugin developers.

rohansingh• 12 years ago

The unfortunate excerpts from this post that stood out for me:

Without access to a lot of source code I can’t tell exactly what is going on, but...
Until Microsoft fixes their compiler to not load their web browser it seems impossible to avoid this problem when doing lots of parallel builds.

In my past life developing applications and services on Windows, this scenario was too common — something is broken, you can't quite tell why because you don't have the source, and you can't do anything but wait for it to be fixed or find some hacky workarounds (which is tougher to do without the source).

This has been the real win on switching to an open source stack for me: you can take your tools apart, look inside them, and fix any issues. I know Microsoft has come further in this in the past few years, but it still sucks to be stuck in that kind of situation.

duncan_bayne2• 12 years ago

Very true. Being able to fix tools and libraries is such a change. I'll never go back to Microsoft (or any other closed-source, proprietary) development if I can help it.

ksk• 12 years ago

May I ask what kind of F/OSS software bugs have you found and fixed (as a user) which you were unable to with proprietary software? I don't need a list, I want to know more about the description of the bug.

NigelTufnel• 12 years ago

Were it GCC instead of VC++ that wouldn't change much for a lot of people. Fixing bugs in a program with 7 million lines of code is not easy.

Fixing bugs in GCC for an unitiated person is probably super hard.

kentonv2• 12 years ago

Fixing bugs in a program with 7 million lines of code is not easy.

It's not like you have to read all 7 million lines of code. The author of the article already has a stack trace; all he'd need to do is skim the functions along that trace. In this case it sounds highly likely that the fix is to comment out some code somewhere that is initializing a resource that isn't actually needed.

Frankly it sounds pretty easy.

bitwize• 12 years ago

This is one of the things about Microsoft APIs that gets me. They all depend on each other in weird ways that have less to do with practicality than they do with getting people on board to use the various Microsoft APIs. For example, DirectX depends on COM. It doesn't have to, but they're both Microsoft technologies so why not? You're going to need to learn how to use COM anyway; it's the future. Systemd is designed this way too; that's why I hate it, despite the fact that it won. It seems architected with a "getting people on board with using certain tools" mindset rather than orthogonally providing functionality.

But this... this takes the dependency tangle to whole new levels of comedy.

Negitivefrags• 12 years ago

COM isn't really a API like DirectX is. It's a model for making APIs. Thus it makes sense that they would use it when creating an API.

I can understand your argument in the general case, certainly, but it doesn't apply very well to COM.

benched• 12 years ago

Long time Microsoft and Windows app developer here. I just thought somebody should point out the obvious: that this is not even remotely unusual for a Microsoft app on a Microsoft operating system. You're just describing the normal architecture of the OS and the way apps work.

jwise0• 12 years ago

I think the true enlightenment in this article (for me, at least) is that Windows's tracing framework is far more powerful than I had realized. Being able to capture a system-level trace is an incredibly valuable tool.

Until recently, I had thought that the only real tools out there to do this kind of analysis lived only on Solaris (DTrace). (As I recall, SystemTap gets part of the way, but does not reach far enough into userspace to really get a good view of what's going on -- is that still true?) It would be very interesting to augment Windows's performance tracing tools with some of the other things that DTrace has to offer -- pervasive low-overhead trace scripting seems like it would have made some of the other subsequent analyses that Bruce was working on much easier.

wging• 12 years ago

Until recently, I had thought that the only real tools out there to do this kind of analysis lived only on Solaris (DTrace).

FYI, there are ports of dtrace to systems other than Solaris.

scintill76• 12 years ago

Sounds like it might be worth it to write a drop-in replacement DLL that doesn't parse XML, or at least in a way that doesn't require mshtml. The XML files that are being parsed appear to be static resources within the analyze DLL itself. A third party might even be able to patch the DLL to have pre-parsed resources instead of XML.

muricula• 12 years ago

If true, this implies that recent versions of IE compiled with this compiler needed older versions of IE to compile themselves. IE is probably the first browser to have the honor of bootstrapping itself.

You Got Your Web Browser in my Compiler

Related Stories