DTrace on Windows

https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902

by ingve • 7 years ago

451 172 7 years ago

techcommunity.microsoft.com

bch• 7 years ago

As a BSD user w partial dtrace support[0], very interested to see what impact this has in the instrumentation world, considering Linux has gone w eBPF[1] for their workalike implementation.

[0] https://wiki.netbsd.org/tutorials/how_to_enable_and_run_dtra...

[1] https://en.m.wikipedia.org/wiki/Berkeley_Packet_Filter

webaholic• 7 years ago

"DTrace has a C-style high level and powerful programming language that allows you to dynamically insert trace points."

Yes, that would be the D programming language :)

adamnemecek• 7 years ago

I think it was worded correctly considering there are two languages called D.

haripatmsft• 7 years ago

yes, that's actually exactly why I wrote it that way.

lallysingh2• 7 years ago

And it's really more awk-ish than C-ish.

vkaku2• 7 years ago

Sweet! Next stop, Microsoft should get officially have ZFS running on Windows (There is a port of it already)

There is some sort of a bitterness in the Linux community when people talk about ZFS or DTrace. But it's the consumer who is affected by it. For their sake, I hope they don't take the 'We have BtrFS/eBPF/X' path.

qalmakka4• 7 years ago

This. They really need a newer, more modern filesystem, especially since Apple has done away with HFS+ and has introduced APFS. ReFS is really dead in the water, and ZFS would give them a powerful, battle-tested filesystem which they could use as a real selling point for Windows Server. Also things such as Windows updates and system restore would benefit a lot from ZFS's native snapshotting capabilities.

Merging DTrace implicitly means they are fine with both the CDDL and Oracle's patents, and thus that ZFS wouldn't be an issue for them.

vbernat• 7 years ago

DTrace has switched to GPL for the kernel part (see https://oss.oracle.com/projects/DTrace/).

fetbaffe• 7 years ago

Microsoft has already developt ReFS for this, however I don't know the adaption rate of ReFS on Windows servers.

https://en.wikipedia.org/wiki/ReFS

Neil442• 7 years ago

Adopting ZFS would take away from Storage Spaces, So sadly I don’t see it happening.

otterley6• 7 years ago

NTFS is in some ways arguably still more advanced than ZFS: think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

Also, NTFS already has snapshots via Volume Shadow Copy.

ZFS is great but it’s not going to be a viable replacement as it exists today.

fxfan• 7 years ago

Wait- how do you find NTFS underpowered? You make vague claims offering nothing of substance.

mixedCase• 7 years ago

bcachefs is profiling itself to finally up the Linux file system game. It doesn't look like it will be upstreamed this year, but it appears to be usable and it's being done by someone who really seems to understand file systems and knows of the trade-offs different FS have made and how they played out.

johannes1234321• 7 years ago

Merging DTrace implicitly means they are fine with both the CDDL and Oracle's patents, and thus that ZFS wouldn't be an issue for them.

Mind that DTrace was ported by Oracle (at least partially) to Oracle Linux first. They haven't done that for ZFS (instead they created bttrfs, before owning Sun)

Sylamore• 7 years ago

I wouldn't say that ReFS is dead in the water, I just think they shifted the focus to supporting virtualization features first to enable S2D/SDN. ReFS is still being developed and enhanced, the work is mostly in server SKU: https://docs.microsoft.com/en-us/windows-server/storage/refs...

bibyte3• 7 years ago

For their sake, I hope they don't take the 'We have BtrFS/eBPF/X' path.

To my eyes at least they are pretty much helpless to do anything about it because of the licences. Do you have any ideas as to how they could work around the licencing issues ?

Edit: I am talking about Linux not Windows.

saghm• 7 years ago

Not suggesting this should actually be done by other distros, but from what I understand, Ubuntu basically just ignored the license and included it anyways

jen203• 7 years ago

The licensing issue for Windows is different to Linux. ZFS as released by Sun was under the CDDL.

The source of incompatibility with ZFS and DTrace on Linux (until recently) was from the GPL, not from the CDDL. Windows (and, indeed MacOS - which has shipped DTrace for years), have no such restriction - the CDDL allows inclusion in larger proprietary works.

kasabali2• 7 years ago

To my eyes at least they are pretty much helpless to do anything about it because of the licences. Do you have any ideas as to how they could work around the licencing issues ?

DTrace has a linux port which is GPL licenced since last year. So they are not helpless and there is nothing to work around.

anth_anm2• 7 years ago

Apparently NTFS is too integrated to replace it.

devchix• 7 years ago

Heh, think about the bitterness in the OpenSolaris community (and the Sun diaspora). Way way back in S10, ZFS, DTrace, zones, and this other weird creature called SMF, they were all going places!

AsyncAwait4• 7 years ago

Sun decided a long time ago it does not want it to be part of Linux, as per its license. It's not us being bitter, I just don't understand why we should get out of our way to accommodate somebody who doesn't want to be on Linux.

Personally, I'd rather see more resources poured into bcachefs[1].

1 - https://bcachefs.org

throw0101a3• 7 years ago

Sun decided a long time ago it does not want it to be part of Linux, as per its license.

[citation needed]

Simon Phipps, Sun's Chief Open Source Officer at the time, has stated this is not the case:

* https://en.wikipedia.org/wiki/Common_Development_and_Distrib...

The only person who has made this claim is Danese Cooper, and she has been directly contradicted by her boss at the time (Phipps).

(Improving bcachefs, or whatever, is independent of the CDDL question.)

snvzz• 7 years ago

The only reason Linux doesn't have Dtrace is because NiH.

Same as with kqueue vs epoll and so on.

seotut2• 7 years ago

I don't understand this attitude. Why the love for Microsoft, after all the abuse it's shown towards other players in the OS market, all the monopolistic behavior, all the ...

There have been better, cleaner, leaner, FOSS operating systems out there for decades, why support, or even give mindshare to something like Windows?

Krasnol4• 7 years ago

Because here in reality Windows is still everywhere and it seems to stay for quite some time. So why not being happy that it's being improved?

Also: https://itvision.altervista.org/why.linux.is.not.ready.for.t...

blablabla123• 7 years ago

Yes remembering this makes me think twice to switch over to Linux again from macOS. On the other hand Windows, BSD and Solaris run on far less devices...

glasslion• 7 years ago

What about an official package management tool first

lelf4• 7 years ago

Kudos for Microsoft for not going the Linux NIH way.

josteink3• 7 years ago

NIH would imply they had a real choice.

Sadly due to licensing, they don’t.

Edit: I stand corrected. GPL should be fine.

lelf4• 7 years ago

It's pure NIH. dtrace is GPLv2. (Also ZFS (CDDL) run fine on Linux.)

grumpydba3• 7 years ago

eBPF was created before dtrace was GPL'd. Oracle is to blame there. 100%.

pjmlp4• 7 years ago

HN loves to hate Oracle, however CDDL license was created by Sun.

benmmurphy• 7 years ago

i think eBPF is theoretically faster. eBPF is verified statically while dtrace loads/writes are all checked dynamically.

i guess eBPF verifier is a bit scary because a small bug anywhere in the verifier creates a big hole. i think project zero found a lot of bugs in the verifier.

its pretty scary putting an interpreter in the kernel. i'm not surprised linux didn't want to have two completely different interpreters in there.

Aissen• 7 years ago

That is one big commit: https://github.com/opendtrace/opendtrace/commit/893932783d5a...

It seems to be on a project which has virtually no contributions since last august.

In addition, I'm not sure where Microsoft is going with this: your DTrace scripts probably won't be portable anyway. The only common point is the language. Is that sufficient ?

jen203• 7 years ago

Depending on what and where you are tracing, scripts likely already are not compatible between different operating systems which support Dtrace - FreeBSD, Solaris and Illumos, and MacOS all have different system calls (for example).

That said, there are _many_ points of commonality which will apply to Windows too - USDT probes, function boundary tracing etc.

bibyte3• 7 years ago

I wonder why they didn't split the commit.

yjftsjthsd-h2• 7 years ago

Do they need to be portable/compatible? If MS wrote their own version/alternative, then it'd be incompatible with everything. This way they get a load of high-quality code for free, and even if it's not directly compatible with existing uses at least the general framework is known by existing users and there's existing documentation so they don't have to do the whole thing from scratch and train all the users themselves. API compatibility would help, but there's value without it.

santoshalper3• 7 years ago

Only time will tell, but it can't hurt to be language compatible. All the unix engineers out there who depend on dtrace will find it more comfortable to use.

On the other hand, using a foreign syntax can hurt too, had they used something that looked more like powershell for example, it might have been easier for Windows admins to get accustomed to.

AJRF• 7 years ago

Do you feel it? The tectonic plates of developer sentiment slowly shifting below your feet. It won't all come in a wave. It is the gradual investment in tools like this that over time amount to a viable alternative to the MacBook Pro running macOS. WSL, Open source .NET, Investing in making Edge work like Chrome (more important to developers and consumers than your ideological quandries with Google owning the web will ever be). The last few years have been great for MS

DoofusOfDeath• 7 years ago

I get the impression that there's a lot of disagreement amongst developers regarding Microsoft's true intentions. I see three main clusters at the moment:

(1) Those who consider Microsoft to be reasonably trustworthy based on recent behavior.

(2) Those who will need Microsoft to continue its current behavior for a longer period before they're ready to trust.

(3) Those who consider Microsoft's current behavior to be at best a partial improvement over the past, or at worst a charm-offensive meant to distract from their ongoing bad behavior. E.g., Windows 10 with unavoidable snooping, and ongoing stealth patent attacks on Linux.

anth_anm2• 7 years ago

The second group doesn't exist, it's just people in the 3rd group lying (to themselves even).

xorcist• 7 years ago

(4) Astroturfing.

Seriously. Everyone else is doing it, why shouldn't they?

pjmlp4• 7 years ago

Windows has always been a viable alternative to MacBook Pro running macOS.

If it wasn't for our iOS projects, they wouldn't be even around the office.

opportune• 7 years ago

Not if you are developing for linux. macOS greatly simplifies that over windows. Of course you can develop for linux on windows but the third party tools that you need to do that are much more complicated and clunky.

zepto• 7 years ago

You mean Microsoft are playing catch-up with a few tools?

That doesn’t make Windows suddenly any less of a mess.

I agree that there are all kinds of issues with Apple’s Mac hardware pipeline, but this is just hyperbole.

saagarjha• 7 years ago

It’s kinda disappointing considering that macOS has always had DTrace and is slowly crippling it :(

jen203• 7 years ago

MacOS X has definitely not "always" had DTrace - it shipped in 10.5 along with the "Instruments" front-end. It's true it is becoming distinctly less useful on OSX though - a shame as I use it extensively as an (application) development tool. This might make me take a slightly more serious look at Windows for laptops in future.

bad_user• 7 years ago

The "Microsoft has changed" is a well coordinated marketing ploy, probably their biggest achievement of the last 10 years. And these days it's actually pretty hard to distinguish the shills from the fanboys that genuinely believe that they changed.

There's something about computing that makes us want to cheer for these big companies as if they are sports teams, no matter how many times they screw us.

But for the uninitiated the biggest problem for some time has been that Microsoft was and still is a big patents troll. They also continued fighting open standards such as the OpenDocument format, even under Satya Nadella. To their credit they did join the Open Invention Network, but it's still unclear what parts of a Linux OS are covered and what aren't. The FSF rightly asked for clarifications on whether the multimedia components are covered and to my knowledge there have been no clarifications thus far.

But forget the years of trolling Android phone vendors with patents such as for the FAT filesystem, here's a story from 2017: http://techrights.org/2017/02/27/microsoft-novell-v2-via-azu...

Also Windows 10 is a piece of spyware shit.

Windows 10 has been intercepting Chrome and Firefox installations and harassing people into using Edge (and this has been my own experience): https://www.ghacks.net/2018/09/12/microsoft-intercepting-fir...

Microsoft is reportedly planning to make Edge indispensable: https://www.theguardian.com/technology/2018/mar/19/windows-1...

Windows 10 has been displaying intrusive ads: https://www.theverge.com/2017/3/17/14956540/microsoft-window...

Windows 10 has been blatantly disregarding user choice and privacy: https://www.eff.org/deeplinks/2016/08/windows-10-microsoft-b...

Windows 10 collects everything you do: https://www.techworm.net/2014/10/microsofts-windows-10-permi...

---

"Investing in making Edge work like Chrome (more important to developers and consumers than your ideological quandries with Google owning the web will ever be)"

That's incredibly naive. First of all they aren't "making Edge work like Chrome", that's an overstatement, the new Edge is going to be only a shell on top of Chromium and nothing more.

Microsoft adopting Edge is either them throwing the glove and admitting that they can't develop a browser, or them pulling another embrace, extend and extinguish. After all, as we've established already, they don't really like that users have a choice when it comes to the browsers they use. This is important to mention, because even for the Chrome fans out there, Edge switching to Chromium provides the perfect opportunity to disallow alternatives on top of Windows. This is just a possibility mind you, when it is far more likely that Microsoft will simply place Edge on life support.

In either case, whether they end up contributing to Chrome's ecosystem, or not, consumers lose for a myriad reasons. And heralding this move as something good is ignorance for the whole history of computing.

endorphone• 7 years ago

Microsoft has had an enormous developer base for decades. Their changes aren't beating competitors, they're simply staunching the flow away from the Microsoft stack. Claiming it as a victory when Microsoft cedes that the enemy has won in a variety of ways is...an odd victory dance. Quite a wave.

I use a Windows 10 desktop for 1/2 my work. The other half I use macOS on a MacBook. They're both great (and the latter is fantastic for virtually any sort of work, whereas with the former I am constantly encountering Windows-specific limitations that demand that I fire up a Hyper-V instance --- WSL is very limited in a variety of deadly ways).

reggieband• 7 years ago

You may be new to "embrace, extend, and extinguish" [1]. Everything looks great during the embrace phase. In fact, everything looks even better during the extend phase. Industry has a short-term memory for this tactic and it does seem to be going in Microsofts favor at the moment.

I'm curious to see how they will act once they regain sufficient market share. Will the old MS appear and will they implement the extinguish phase?

1. https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

ocdtrekkie• 7 years ago

Strategy was uttered in 1995 by someone who left Microsoft in 2000. It is 2019, and this phrase is not "new" to anyone. Please stop doing this.

Iwillgetby3• 7 years ago

Can any microsoft folks discuss similarities/differences between dtrace and procmon? Besides the obvious gui vs commandline difference of course.

Is the performance on par with running dtrace on solaris?

Birch-san• 7 years ago

the scope of DTrace far exceeds that of procmon. procmon will show you params and return code for file accesses, and provides filtering, but that's the limit.

whereas DTrace could tell you "these are the file accesses that took longer than 5ms" or "here's a histogram/heatmap of file access durations". DTrace can correlate — file accesses by a given process in-between a given function's entry and exit — telling you _why_ a file was accessed.

well, maybe you're just asking how performant DTrace would be at achieving the same output. sadly I can't answer that.

polskibus• 7 years ago

I wonder if they backport dtrace so it can be used on older Windows Servers (2012 for instance)? Or maybe they can somehow make it work in .NET Core only for such operating systems - it could still be very useful.

lelf4• 7 years ago

Unlikely. Dtrace needs support from many OS components, including kernel.

santoshalper3• 7 years ago

Not too likely. The hard part of implementing dtrace isn't porting over the program itself, it is the huge number of OS kernel hooks required to instrument. Those are unlikely to be back-ported. The really interesting question is now that they have the hooks, will they consider making additional interfaces with their own languages (powershell anyone?)

Iwillgetby3• 7 years ago

Good dtrace stuff here that shows the power of dtrace on Solaris, MacOS, and FreeBSD.

http://www.brendangregg.com/dtrace.html

brendangregg• 7 years ago

Congrats to Microsoft, I look forward to running my old DTrace scripts on Windows!

This also means they have done a lot of the platform work needed to support eBPF one day, to really catch up to Linux.

Iwillgetby3• 7 years ago

Dtrace is amazingly powerful. Thanks for all your contributions over the years.

http://www.brendangregg.com/dtrace.html

cryptonector4• 7 years ago

Would you say eBPF is better than DTrace now?

chupasaurus• 7 years ago

To run native containers, they've implemented almost all Linux kernel namespaces (IIRC mount was almost there already) and resource limiter. So, if there are some profits on the horizon, that may become a reality.

tambourine_man• 7 years ago

I love this new dev friendly Microsoft.

Makes me more enticed to switch everyday.

tracker13• 7 years ago

I'm happy to see the changes, and more than happy to consider Azure and even like Azure Devops... I'm still a bit more comfortable on a Mac or Linux environment. There are some odd windows-isms for bash on windows, or the LSW, or Docker Desktop, etc.

I'm hoping a lot of it gets more solid in the next couple years. As it stands, I'm happier to see .Net Core running outside windows more.

GuB-42• 7 years ago

Microsoft has always been dev friendly.

What's new is that they now acknowledge the existence of the UNIX world and open source.

j16sdiz2• 7 years ago

Not "always". It is not long ago when Visual Studio get community edition.

bsder• 7 years ago

Translation: Microsoft needs DTrace to debug and control Azure.

santoshalper3• 7 years ago

I'm sure scratching their own itch was a significant part of this. DTrace is fantastic, and Microsoft is one of the largest server operators in the world so the need is clearly there.

cryptonector4• 7 years ago

Anyone operating a cloud of remotely interesting size will need DTrace, or, if they use Linux, eBPF tracing. It was a foregone conclusion that Microsoft needed DTrace.

Microsoft surely also needed it for other reasons. Developers, sysadmins, and power users -- all need DTrace or DTrace-like tools.

We're finally leaving the dark ages of unobservable operating systems! Hooray!

We should all give huge thanks to the creators of DTrace, Mike Shapiro, Bryan Cantrill, and Adam Leventhal, as well as Brendan Gregg and all of the crew at http://dtrace.org.

renholder2• 7 years ago

Only available on x64 Windows and captures tracing info only for 64-bit processes

Does that include Wow64[0]?

[0] https://docs.microsoft.com/en-us/windows/desktop/WinProg64/w...

josteink3• 7 years ago

As far as I know wow64 exists to run 32-bit processes on 64-bit Windows, so given their statement I would assume no.

renholder2• 7 years ago

WOW64 is a 64-bit "wrapper" for 32-bit processes. You can see this when you debug a WOW64 process in WinDbg[0]; so, the system is agnostic about it's "bitness" because it "appears" to be an x64 process but it's really 32-bit, just wrapped. Thus, the posit. :)

[0] https://docs.microsoft.com/en-us/windows/desktop/WinProg64/d...

ksec• 7 years ago

At this rate I would not be surprised if Microsoft Open Source part of the Windows Kernel in future.

AndrewDavis• 7 years ago

I'm curious if this will be available on the Linux subsystem for windows?

Anyone have an insider build to test?

cryptonector4• 7 years ago

All those applications that added USDT providers... that work will pay off even more now!

DTrace on Windows

Related Stories