November 2007 - Posts
After several repetitions and preparations, Alon and I finally presented our level-300 session introducing the Windows Server 2008 operating system and the new mechanisms which should be interesting for developers. We would like to thank everyone for coming to the session (by the way, the video should be available for download from Microsoft's web site shortly). Presenting together wasn't an easy task, but I feel that the coordination and dynamics between Alon and I were beneficial to our presentation.
Take a look at the DEV323 Session Slides and Demo Code to see what mechanisms you're missing if you haven't tried Windows Server 2008 yet. By the way, you can always download the latest release candidate (RC0) and give it a try (unless you attended the session, in which case you should have a DVD ready for installation and evaluation).
Finally, if you're interested in the underlying APIs and missed Alon's "deep dive" (DEV424) session for some reason, watch his blog to stay tuned for the session slides and demo code.
[Updated: Dec 19] If you missed the lecture, you can always tune to the video recording.
A few hours after my level-400 presentation on performance and debugging, I think I can conclude that it went great. I would like to thank everyone for coming to the session; and for those of you who didn't have a seat available and went to see another session - hopefully soon enough the video will be available for download from Microsoft's web site.
For now, you can take a look at the DEV409 Session Slides and Demo Code (in Office 2007 format; you can use this KB article to view the presentation in previous versions of Office). As promised, all the funky WinDbg and SOS commands that I have used are listed at each demo slide.
The primary takeaways from my session, IMHO, should have been:
- Know the theory - there's lots of theory behind debugging in general and .NET debugging in particular; without this theory you will be strolling in the dark. It's almost compulsory that I shamelessly advertise the 3-day .NET Performance course I'm teaching for Sela, featuring an in-depth dive into the theory and practice of .NET internals such as the type system, the garbage collector, and much more.
- Know the tools - it's almost impossible to debug anything nowadays without the proper tools installed and configured; WinDbg and SOS are a very powerful addition to your arsenal, but there are also many more.
So thanks again to everyone who attended the session, and I look forward to seeing you all at the next Microsoft event. I would also like to use this opportunity to thank Microsoft for hosting this day; I feel that many lessons from the last Developers Academy have been learned.
Hey, we're inside MMC and we're looking at this post!
I prepared a deck of slides for the upcoming Developers Academy (November 27, Airport City).
You can find it at the following link, the ZIP is approximately 4.75MB and it contains a PowerPoint presentation in Office 2003 (PPT) format. (I wasn't able to upload it to the blog, so I resorted to a third-party file hosting service. Please bear with me/them for the annoying 30-seconds delay, and the possibly slow download speed.) This download link will become unavailable in 14 days (December 6).
[Updated: Nov 24] There was something wrong with the upload service I used. Please use this link for the download (still 4.75MB with a PowerPoint presentation in Office 2003 format).
[Updated: Nov 27] The final deck of slides presented at the actual session, including the demo code, is now available to download here (Office 2007 format this time). Also note the post announcing this download.
[Updated: Dec 19] If you missed the session, you can always tune to the video recording.
Please note that this is not the final set of slides I am about to show at my session; instead, it's just a teaser for those of you considering to join me. After the session, the presentation, demos and video recording of the entire thing will be available on the web.
The session code is DEV409. It will be held in the Golan room (floor -1) at 16:15, just after Alon's "deep dive" lecture on Windows Server 2008.
The demos I will be showing will focus on a sample application experiencing the issues outlined below. We will be diagnosing, debugging and solving these issues throughout the session.
You get a phone call in the middle of the night because ...
-
Once in a while, a memory leak begins to manifest under heavy load on the application.
-
Under even heavier load, the application spends too much time in the GC, hindering the performance of critical business processes.
-
After a certain amount of time, the application hangs without processing any further requests. (Obviously this would always happen in the middle of the night, during the weekend, when you're on vacation with your family and all the other developers are abroad and unable to support this.)
-
The day before you just checked-in a change where you removed a few unused legacy fields from a commonly used data structure. Suddenly the application runs 25% slower than usual.
-
Adding additional processing power to the system didn't increase its overall throughput. You're using multiple threads, and are using them correctly. What the heck?
(By the way, feel free to comment or contact me through the blog's contact form if you have any other interesting scenarios.)
This post really got me. The writer describes a system design conversation, when one of the arguments is that inheritance in .NET is a possible cause of performance degradation. The author then continues to test this "degradation" using the following setup (deduced from the original post):
class Animal {
private int _legs;
public virtual int Legs { get { return _legs; }
}
class Dog : Animal { }
...
foreach (Animal a in animals) {
int i = a.Legs;
}
foreach (Dog d in dogs) {
int i = d.Legs;
}
He then thoughtfully concludes that inheritance has no performance impact, in view of his measurements.
Let's just ignore all the issues concerning the actual value of this kind of measurement, and try to focus on the original statement. Why would accessing a property on an object be affected by the inheritance chain of the object's type?
To really understand this kind of thing, you need to have virtual method dispatch memorized by heart. It's not rocket science really, but few developers (especially .NET developers) are aware of it. Consider the JITted code generation for accessing a property on the Animal object as opposed to accessing it on a Dog object, in this simplified scenario:
Animal a1 = new Dog();
int i = a1.Legs;
Animal a2 = new Animal();
i = a2.Legs;
...
0000001f mov rax,qword ptr [rbx]
00000022 mov rcx,rbx
00000025 call qword ptr [rax+60h]
...
00000042 mov rax,qword ptr [rbx]
00000045 mov rcx,rbx
00000048 call qword ptr [rax+60h]
(Note that this is 64-bit assembly language; it's not essentially different from 32-bit, so I didn't bother forcing a 32-bit environment.)
I've snipped everything irrelevant to the discussion. Note how absolutely identical the code is for both properties. If you have the theory behind you, it's also absolutely obvious why this is so - the method table entry for the Legs property will always be at the same offset for any derived type of Animal. You can continue the derivation hierarchy forever; it will still be the same code to access the property, and the same slot in the method table.
What if the property discussed in the original post was not virtual? I considered leaving this as an exercise for the reader... But here goes. Accesing a non-virtual property as trivial as this one is obviously going to be inlined. The property's backing field is stored in the Animal class part; if we have a Dog object, it must be at the same offset. Here's the relevant code generation:
...
0000001f cmp byte ptr [rbx],0
00000022 mov eax,dword ptr [rbx+8]
...
0000003f cmp byte ptr [rbx],0
00000042 mov eax,dword ptr [rbx+8]
Surprised? By now you really shouldn't be. The property is stored at the very same offset in both cases, and therefore the inlined code to access it is absolutely identical. (Theoretical side note: the first instruction in each pair merely enforces a null reference check, to ensure that we are not calling a method or accessing a property on a null reference. Note that we don't have to do anything with the result of the comparison - it's the very fact that we dereference the reference, as in [rbx], which would cause an access violation, which would be trapped by the CLR and converted to a NullReferenceException.)
So what can we conclude from this brief analysis? That comparing a piece of code to an absolutely identical piece of code will yield identical performance for both pieces of code. Stunning, really.
Sometimes there's this point in debugging an issue when you can't take it anymore. You've tried to diagnose the *** from every possible perspective, try out various configurations, use brute-force binary elimination, whatever. It just doesn't help. And as it often gets with these particularly nasty bugs, it only gets worse over time. Was it easy to reproduce it when you started? It's no longer easy. Could you work offline on your own machine and fix the issue for the customer? Now it works smoothly on your laptop; this hardly makes the customer happy if it doesn't work on his production server.
This is the case with a bug I'm hunting right now. Figured I might just as well post it here for the minute chance someone might have a valuable contribution. (Note that I am unable to disclose the customer's identification information, or any debugging-related information gathered from the customer's machines.)
Basically, the customer's issues started with unexplained application crashes (ExecutionEngineException, if I may add) with a bizarre call stack indicating what appears to be an access violation while JITting a piece of code. Granted a giant code-base of over 160 Visual Studio projects, it took a fair amount of time to obtain a simple repro which can be separately dissected. It was even easier to obtain that repro since we established that the same assembly which caused the exception during JIT-time, was also causing an exception during NGEN time (which doesn't require any setup to perform). In NGEN, the exception was slightly different (some mumble about RPC call failing), but the call stack up to that point was basically the same. Another noticeable detail was that ILDasm had no trouble whatsoever opening our assembly; on the other hand, PEVerify cried like a little girl and instead of saying something about invalid IL, it simply crashed with a similar call stack!
Having obtained an accurate repro of the problem, we tried to reproduce it on our own machines. We tried .NET 2.0 out of the box, we tried .NET 2.0 with VS2005 SP1, we tried all kinds of hotfixes issued by Microsoft over the years. We even tried compiling with the "updated" compiler and NGEN from the .NET 3.5 Beta 2 release. What we didn't succeed at was reproducing the problem.
Frustrated as hell, we approached the customer again, with some fresh ideas. It seemed that when compiling the assembly from Visual Studio, the problem would occur; however, if compiled using the msbuild utility from the command line, the problem would not occur and everything was happy ever after.
Again we started a quest to determine what's wrong with that Visual Studio installation, to no avail. I was desperate enough to fire up Process Monitor and analyze the differences between what the C# compiler (csc.exe) is trying to do. At this point I was hit by a surprise.
Visual Studio did not even invoke the C# compiler; it did try looking for it in the usual (C:\Windows\Microsoft.NET\Framework\v2.0.50727) directory, but it didn't use it for compilation. However, it still managed to produce an assembly, albeit invalid; appearingly without the aid of a compiler.
After a little bit of web investigation, it appeared to me that Visual Studio 2005 has a curious performance optimization: unless otherwise specified, it doesn't actually invoke the C# compiler (csc.exe) or the VB.NET compiler (vbc.exe); instead, it uses an in-process hosted version of the compiler. This behavior can be controlled by inserting the <UseHostCompilerIfAvailable>false</UseHostCompilerIfAvailable> tag into your MSBuild project (you can do this for the Microsoft.Common.targets file, or for the .csproj file in question).
Using this tag had the amazing effect of actually "fixing" the problem; the out-of-process standalone compiler now produced correct, verifiable code.
At this point I was happy enough to let the problem and myself go home. However, the customer reminded me that the original issue was brought up by QA engineers, who were not using a version of the application compiled in Visual Studio. They were using a version from the Team Foundation build server. Which uses msbuild. Which directly invokes csc.exe, with no "hosted compiler" babble.
Exhausted as I was, we tried to reproduce the problem on the build server. To make a long story short, it didn't reproduce. Everything worked with msbuild. Everything worked as part of the TF build. Everything worked from Visual Studio, with and without my "host compiler" patch.
Magic. Back to square one.
A terribly short and out of band post to attract your attention to the Developer Academy II, taking place at the Airport City (Israel), on November 27.
I'll be conducting two sessions: Psychic Debugging and Performance 101 (400 level), and Windows Server 2008 (300 level). The latter session will be a joint one with Alon Fliess (Sela CTO and VC++ MVP). I might also hold a quick jam session on native debugging/troubleshooting, but it's undecided yet.
It's still hard to determine exactly what I'm going to be covering in these 75-minute sessions, but here's a preliminary lists of thoughts for what I had in mind...
Psychic Debugging and Performance 101 - excite developers and familiarize them with debugging and performance tools and techniques. This will likely be a case study story covered from various perspectives, such as troubleshooting .NET memory leaks, diagnosing excessive GC, diagnosing and debugging hangs (e.g. deadlocks), obtaining a post-mortem crash dump and analyzing it, and analyzing a performance issue of choice.
Windows Server 2008 - we will be focusing on demonstrating the mechanisms of the new server OS, attempting to prove that almost any application will benefit from the move to the new platform (even without any modification). We'll be covering mechanisms such as the Restart Manager, Kernel Transaction Manager, various I/O and networking optimizations, and lots of other things new to Vista/Server 2008.
(By the way, Alon will be conducting a separate level 400 session diving deep into the new APIs which we will only briefly mention in our joint lecture.)