A while ago I started a project to add DTrace probes into common open source applications we use at OmniTI. I added them into Apache/APR-utils (which were put back) and PostgreSQL (which were also put back). I’ve also added metrics exposure to the many applications we write internally (either in JSON or the Resmon XML DTD) over HTTP. It’s so easy these days to include an HTTP server into whatever daemon process you are writing/working on that there are almost no excuses to not wire one up and expose your application’s internal metrics.

Continue reading

DTrace and Apache

DTrace kicks ass. Everyone knows I love ZFS, but in my opinion DTrace is the most significant and disruptive systems technology in at least 10 years. Today I gave my talk at ApacheCon; it was well received. I hope that I inspired some people to go use DTrace and look in their systems' underpants. Here are my slides and the scripts I used for my live demo. I'll update this entry with the video as soon as I have the link.

Continue reading

I met Bryan Cantrill a few years back at OSCON where he gave a session on DTrace. As any reader of my blog would know, I'm a big DTrace fan. Bryan's presentation style was passionate and animated and I was quite convinced there was an illegal substance involved. I've come to learn that style is Bryan's style. Since then, I've had a few opportunities to dialogue with Bryan and on every occasion he reinforced my opinion that he is one of the more insightful minds in today's computer engineering field.

Continue reading

I recently attended dtrace.conf(08), which was a blast, but I left that conference with a single thought and it has been reinforced since. Everything should be dtrace enabled. While it is true that using DTrace you can introspect just about everything in the system, the pid provider (used to trace inside user-space applications) requires the user to to know the code of the application. A full system "in-flight" has too many different apps running for me to keep all of their code-bases in my head.

Continue reading

I am oft in the position where SCSI is not working right for me. Whereas I always seem to be in the position where SATA over promises and under delivers. When SCSI misbehaves, it usually really matters. Where's my disk go? Time outs? And the king of all frustrations: wassup with that tape robot?! This clearly puts me in the sysadmin geek category, but I had to share. I had a spin-tingling euphoria when I read the details of SCSI packet sniffer written in dtrace.

Continue reading

As many people already know, I'm a big fan of DTrace. Well, today I attended dtrace.conf(08) -- the first (un)conference revolving around planet DTrace. It was awesome. Many people who know me well have heard me say, "my good days are when I'm the dumbest person in the room." That's not to be confused with "I like having bad days." Instead, I like to be at my best and still struggle to keep up.

Continue reading

We've been doing a lot of PostgreSQL work lately and we have one largish system (terabytes) that runs on top of Apple XServe RAIDs. While people argue that SATA is getting better, let it be understood that Fibre channel SCSI drives rule. The difference between carrier class storage and "enterprise" (a.k.a. commodity) is pretty tremendous. While this system will eventually make good use of the XServer RAIDs and long-term storage containers for write-once read-many data tables (archives), the "

Continue reading

Author's picture

Theo Schlossnagle

Distributed Systems, Scalability, and Operations. read more

CEO - Circonus

Maryland, USA