Hello from OSCON. I gave my full-stack introspection crash course talk today. It has been quite a while since I've presented anything in a 40 minute format, but I think the talk went quite well. I got a lot of positive feedback.
I decided to take a risky approach inspired by dtrace.conf(08) by demonstrating dtrace on a live, mission-critical system we run at OmniTI. The risks of this are: network connections flake out, dtrace doesn't work correctly or I do something stupid and cause some service unavailability. Well, as I use dtrace just about every day, I wasn't worried about the breaking things. And while my network connection winked out for about one minute and dtrace has some annoying sub-second aborts, I think the demonstration was quite effective.
Many people gave positive commentary at the end and afterward. People asked for the slide to be put online... and while they have no real content of value (as the demo was everything), I put them here anyway:
In addtion to the slide stack, I've included the simple scripts that I used during the demonstration. I ran qps.d, query_speed.d, and query_speed2.d on the database server and r3.sh, perl.d, pcpu.d, papcpu.d, and papcpu2.d on the web server. These require The Devel::DTrace perl module, PostgreSQL patches, and Apache 2.2.8 patches. Some of them are approximations of correctness, so weigh the output appropriately (the perl ones). Enjoy!