Comtypes: How Dropbox learned to stop worrying and love the COM

https://tech.dropbox.com/?p=393

by frsandstone • 14 years ago

91 34 14 years ago

tech.dropbox.com

snprbob862• 14 years ago

COM is a necessary evil when integrating with Windows, Office, Visual Studio, or any other big/old Microsoft product. That's just how it works.

The .NET framework actually does a pretty good job of hiding the complexity from you, but having done some serious integration with VS, let me say that's an abstraction leak that I wouldn't wish upon my worse enemy.

gecko3• 14 years ago

Where have you found the abstraction leaky? I've generally found it works just fine; the only leaky part of it I've found is in some of the Microsoft-provided .NET wrappers, which I've seen leak memory. (Directly using the COM objects works around that 100% of the time for me.)

Edit: I'm not really being clear above. When I'm saying I've found the leaky abstraction to be around memory management, I'm not conflating "leaky abstractions" with "memory leaks," except insofar as the big point where I find problems with COM is around the need to be much more explicit with resource management. That the leaky abstraction happens to be around the...er, leaks, is coincidental.

dljsjr• 14 years ago

Leaky abstractions have nothing to do with leaking memory. A leaky abstraction is a layer that attempts to reduce complexity and fails at its job by not hiding enough of the ugly parts.

It's a Spolsky-ism: http://www.joelonsoftware.com/articles/LeakyAbstractions.htm...

Edit: Egg meet face.

snprbob862• 14 years ago

The average consumer of Microsoft code instantiates a few objects and calls a few methods. That, for the most part, works extremely well. Memory management, as you mention, is a big pain point because you immediately are faced with reference counting. However, by itself, that's not a big issue.

Things go crazy once you decide that you want to give your COM object to somebody else in a framework-style callback pattern. For example, if you want to extend Visual Studio, you need to implement a bunch of interfaces, manage memory yourself, deal with object registration and GUIDs, worry about invalid interfaces and versioning, etc. And that's just all the standard COM stuff, that you're faced with 100% of the time.

But there are compounded complexities. COM uses HRESULT return values, .NET uses exceptions. Most public interfaces quickly start looking like dynamicly typed code: full of references to IUnknown and QueryInterface calls. You need to understand all the threading apartment model stuff, but the .NET libraries bundle up some best practices there that often don't play nice if you need to slightly bend the rules.

Finding documentation is impossible because the web is dominated by low quality questions with low quality answers using the same terminology. Using the Microsoft personas, the Morts and Elvises drown out the Einsteins in the Google results. However, even having the VS source (which I did), it's far too big to grep and far too many levels of QueryInterface indirection to grok.

Need some enum values? You'll need to redefine them yourself. Go find the C headers and copy out the values.

Need to deploy a new version? Good luck understanding the rules of AssemblyVersion, AssemblyFileVersion, the class loader, Locales, etc. Some random error happening somewhere in the class loader? You need to install some little utility program that some guy in DevDiv made that makes a secret registry key set up some secret logging to some secret IPC channel. Argh.

I was tasked with wiring up deploy/debug for WinPhone 7. I needed to implement a few dozen methods from the IVsDebugger interface, do all my own threading and networking, and then spend a few weeks debugging synchronization issues with the UI. Meanwhile, I had the exact same task for Familiar Linux on a Compaq iPaq several years prior to that. Despite Eclipse's bloat rivaling Visual Studio's, all I needed to do was change a config file from `$PROG $ARGS` to `ssh $HOST gdb $PROG $ARGS`.

EDIT: Just wanted to add that I found this post difficult to write. I, for the life of me, couldn't recall the interface name IUnknown. In the 3 years I've been gone, I've forgotten more about COM than most .NET programmers will ever know. I'm quite happy about that...both for me escaping, and for those .NET programmers lucky enough to remain ignorant of these things. In general, all of DevDiv does a great job keeping this complexity away from people writing line of business applications.

dmethvin• 14 years ago

I never thought the .NET Framework did enough complexity hiding to justify a 60MB installed size, especially if it was just providing a bit of plumbing to get you back down to the COM level.

Most COM interfaces support late binding, so you can easily call them from a language like JavaScript, which is already installed in Windows as part of the platform. Ten years before node.js existed, you could run this JavaScript on Windows 2000 to read and print a file:

    var fso = new ActiveXObject("Scripting.FileSystemObject");
    if ( fso.FileExists("myfile.txt") ) {
       var ts = fso.OpenTextFile("myfile.txt", ForReading);
       WScript.Echo(ts.ReadAll());
       WScript.Quit(0);
    } else {
       WScript.Echo("File Not Found");
       WScript.Quit(1);
    }

tedunangst2• 14 years ago

The .NET framework is more than just a bit of COM plumbing.

dguaraglia• 14 years ago

Well, that's a facetious argument if I've ever seen one :). .NET does a lot more than just interface with COM or the Windows API. VB (pre .NET days) had much better integration with OLE/COM and the runtimes were like 1mb in size at worst.

I've always thought of COM as one of those almost brilliant pieces of technology that somehow got completely ignored. Of course, it's still commonplace in the Windows world, but the concept facilitates reusability so much, it's a shame it wasn't adopted by other platforms.

achal• 14 years ago

Semi related: Not sure how many people have played with PowerShell/COM, but it's fun to toy with. Haven't used it for anything too useful (yet) but for example, to delete all the comments in a Word document in a couple lines:

$a = New-Object -com Word.Application

$a.visible = $false

$a.Documents.Open("{absolute path}").DeleteAllComments()

Better examples: http://www.simple-talk.com/dotnet/.net-tools/com-automation-...

eckyptang• 14 years ago

And this and VBA is why office is so irreplaceable. I wrote a report generation tool in 1999 with word 97. It picked a report definition off a file share, assembled documents from fragments on disk (up to 50 pages a pop with tables, graphs etc) and emailed them or printed them. 2000 documents an hour on a single Pentium pro 200 with 128mb of ram on NT. It took 2 days to write and was used until 2009 by 9000 users

This was just word, com and nothing else. Even the application host was an instance of word.

I doubt it could have been done with anything else then or now.

ta121212• 14 years ago

Provocative title for a pretty generic article.

1. Summary of COM, a widely used technology

2. Summary of comtypes, a python package for interacting with COM

3. One example of a gotcha they ran into with comtypes and arrays of COM objects.

Conclusion: meh.

ejdyksen2• 14 years ago

How is the title provocative?

owlishness2• 14 years ago

EJ...is that you? What a coincidence. What's up, old office-mate?

ta121212• 14 years ago

The title, "How Dropbox learned to stop worrying and love the COM" is a reference to the film "Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb". The object taking the place of the atomic bomb in the variant title is COM. To me this implies that the article will describe why people worry about COM and why they're wrong. I did not find it to contain either.

Admittedly I have never seen the movie, which wikipedia describes as a "dark comedy satirizing the nuclear scare". I suspect the article doesn't follow along those lines either.

bugsbunnyak2• 14 years ago

Am I reading this correctly? they are using comtypes as an exploratory tool, but the photo upload feature was written in something else...

(I haven't disassembled the dropbox dlls, but there aren't any obvious python signatures in the install directory)

If the above is wrong, I would love to know what they are using!

owlishness2• 14 years ago

No, Dropbox is using Python and comtypes in its actual product.

bugsbunnyak2• 14 years ago

Any idea what they are using to package the Python into the exe?

I've tried py2exe and all of the similar packagers that I could find, and every one of them requires brittle incantations to get everything packaged (I'm using a lot of graphics and number-crunching code beyond standard python, but all from the package index). I guess on the scale of Dropbox this is probably not a big issue. On my scale (1), it's one of the reasons I've started using C# (an unexpectedly awesome experience so far, btw). I need the fastest possible path to one-click exe files that non-technical (bio/chem) researchers and RAs can run. I've belatedly realized that the overhead of doing this in python is really too high.

[edit: found this SO answer: http://stackoverflow.com/questions/2678180/how-does-dropbox-... but it doesn't give specifics. In particular, AFAIK, none of the packagers can eat libpython.dll into a larger exe files]

[edit: interesting - http://blog.codepainters.com/2012/09/17/python-care-and-feed... ]

cek2• 14 years ago

"COM is Love" - Don Box [1]

[1] http://en.wikipedia.org/wiki/Don_Box

mahmud• 14 years ago

What do you expect, he wrote the book on it.

cek2• 14 years ago

And I wrote the forward to the book on it :-)

tonyedgecombe• 14 years ago

If you read Don Box's book on COM he takes you through each part of it explaining in detail why it is the way it is. Every point is justifiable and the book is convincing.

However it clearly didn't work as a whole as Microsoft felt the need to replace it with .Net for application level programmers.

dpark• 14 years ago

>So, the following code

  for i in range(100):
      print idevice_item_array[0]

>actually crashes Python because it creates and destroys a hundred IDeviceItem objects, resulting in a hundred calls to Release on the real COM object. After the first Release call, the COM object is considered deleted.

This doesn't sound like a "gotcha". It sounds like a huge bug that comtypes needs to address.

Comtypes: How Dropbox learned to stop worrying and love the COM

Related Stories