OpenSolaris

Reconnoiter is coming along. Unlike most open source project, I tend not to talk about mine until their are really useful to people. Over the last year, I've adopted the unhealthy attitude that useful means "shiny front-end." So, I'm blogging to break that attitude and talk a bit about project that doesn't have a shiny front-end... yet.

Reconnoiter is built out of years of frustration using tools like RRDTOOL, Munin, Cacti, ZenOSS, Nagios, etc. etc. I have a lot of problems with these tools. First, they are not efficient. I need a powerful machine to monitor a mere 10k services. And it actually gets to be an engineering challenge to monitor 100k services with these tools. Also, the graphs are about 10 years old with respect to design and usability. I want something new, something fresh, and something that doesn't need a damn web UI to configure. Several people have asked, why are you reinventing the wheel? Why don't you just improve an existing product? My answer is that I want a well-thought-out product foundation so that I can trust all the bits. I want reponsibilities decoupled at the right spots. I want data in a form that the world can query and run reports the likes of which I have not concieved. I don't want the load on my monitoring machines to be 8. I want my monitoring system to check services and metrics when it planned to, not several minutes (or event 2 seconds) after it told me it would. Simply put, I expect it to work well, all the time. And, of course, I want it to work how I would expect it to work.

Reconnoiter was born out of the need to monitor the internals of many disconnected data centers with between 10 and 1000 machines in each facility. Monitoring can mean a lot of things, here I consider it to be the collection of metrics and awareness of their availability. In and of itself, monitoring is pretty useless, but it is the foundation for two critical pursuits in Internet infrastructure and business management: fault detection and trending.

Fault detection is as simple as understanding when something has faulted. However, knowing something is broken is easier than knowing something is about to break. Is it better to know that your machine just crashed because the chip slagged to the motherboard, or that the temperatures in rack 043 are rising unexpectedly? Answer: both, but I hope I only learn the latter and not the former. Truly, there are too many things to monitor... hundred or thousands of metrics on each piece of equipment. I can't reasonable go in and configure good/bad thresholds on each one. I want anomaly detection. I want a system that I can say: "this looks right, tell me when it stops looking right." That, to me, is a much need companion to tradition fault detection.

To me, trending is much more than drawing graphs... it is about intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you fucked up getting where you are now -- in the vain hope that you learn from your mistakes and plan better next time.

Reconnoiter is an attempt to build these things. Building a system requires starting with pain (need), solid structure and plumbing (good engineering). So, reconnoiter is underway. And this post is in mid-step:

It started on OpenBSD, and added support for FreeBSD, Mac OS X, Linux.

As of changeset [292], we have Solaris/OpenSolaris support.

We have a pretty nice front-end for trending under construction, but it isn't there yet. We'll have numeric data combined with textual "event" data on the same graphs. All that convenient stuff. Here's the rather plain-Jane graph you get now (because some people won't even read a post if it doesn't have a pretty graph):


Honestly, I don't know what the value of this post is, but people around here keep telling me that people should be aware of an open-source tool like this, even if it isn't finished (read: usable) yet. I say it isn't usable yet, but on our development instances here, we monitor 2892 production metrics across two data centers and the load never peaks past 0.10. I'm pretty excited about where this is going. Honestly, my favorite part right now is that I can configure and control the noitd checking nodes via a telnet console and it acts as if it is a piece of network equipment rather than an "application" -- as it should be IMHO.

Today someone asked me: "You speak about ZFS a lot. I know other people that talk about the latest filesystems with praise, but generally speaking they just don't have much to offer. Is ZFS that different?"

My answer is "yes." But, of course, I can't leave it at that. I'm not going to make a performance argument -- ZFS is fast in some cases and slow in others -- just like everything else. I think one of the things we've seen in the last 10 years is that everyone felt the need to come out with their own filesystem -- at least on Linux. So, you have to as yourself why. My personal opinion is that filesystems on Linux suck.

Most filesystems on the market support snapshots. No open source filesystems on Linux (that I'm aware of) support snapshots. Of course, you can use LVM to do block-level snapshots. First off, that's a pain in the ass w.r.t. storage provisioning. Other systems make the process of allocating and managing snapshots "not my problem." (simple and easy). Let's be frank, ext2 and ext3 are nothing to write home about. reiserfs, xfs, jfs, the list goes on and on.

There are a few closed-source filesystems that are really nice. Specifically Veritas Filesystem (VxFS) and its excellent layered volume manager VxVM which appears to have heavily inspired geom on FreeBSD. DEC thought it was so cool that they pulled it white-label into Tru64. Respect.

So, what makes ZFS so different? ZFS is a disruptive technology as it abolishes the sacred line in the sand between block devices, volume management and filesystems. This means it just make storage management easy. When I say easy... I mean easy.

So you want more space? Add more disks. Want to move from from failing disks to replacements? Tell zfs to add the new ones and tell it to remove the old ones. Read that report by Google about disk errors? ZFS checksums all data. My personal experience says checksums are good. Snapshots? Sure snapshot to your heart's content. We snapshot some systems hourly and never ever delete the old ones. Snapshots are really cool, but what if you could rollback to a snapshot? zfs rollback. What if you wanted to make a read/write copy of the fileystem or an old snapshot? zfs clone. You want to store a lot of raw data? zfs has built-in compression. Oh, and it is open-source.

Simply put. ZFS. Respect.

For those interested, here is my slide stack from PostgreSQL Conference East '08. I think the title of the talk was "PostgreSQL: Looking under the hood with Solaris."

The presentation was 90 minutes long and had lots of shell-based show-and-tell. Obviously that stuff isn't available in the slides. I think it went over quite well. The audience was small, but hopefully people took away the a lasting impression of what DTrace has to offer and at least one person had the response: "By the power of Greyskull." Regardless, enjoy the slides!

New acquaintances walk away from their first conversation with me and either think that I am in love with a particular vendor or technology or they think I truly hate all technology. Both are true in some fashion.

The fact that I have an OpenSolaris feed on my blog might indicate that I'm a fan of Sun. The truth is I am and I am not. As is true of any large organization, it's really tough to be enamored with all of it. I am a huge fan of Solaris 10 and Sun's initiative to support strict ABI compatibility for stable interfaces, and I'm downright giddy about their ZFS and DTrace technologies. I think Zones/Containers are cool and I think their engineering team has some brilliant shining stars and is on the whole smarter than average. Yet, the OpenSolaris community is challenged in a lot of ways due to the corporate involvement by Sun that leaves me with a funny taste in my mouth. I'm luke-warm about Java and feel like their hardware initiative is going down-hill with bad quality problems on some of their new offerings compared to their spectacularly rock-solid history (sans the E4500).

Recently, I did an interview with Mark Thacker of Sun about our use of Solaris for their Solaris Podcast series. We've had some bad experiences here and there, but all-in-all it has been a win. DTrace has been a god-send and ZFS has saved my bacon several times. Anyone who's talked to me knows that I'm brutally honest and appreciate those that return the favor. I'll look at solution and I tell you what you did wrong. I don't tell you what you did correctly... after all it was all supposed to be done correctly. So, upon listening to this Sun podcast, several of my colleagues said: "that had to have been edited." As marketing people usually do, they attempt to limit the negative exposure as much as possible -- most notably, they removed a section about lack of tight integration between ZFS and Zones which has made for some very painful upgrade paths. We have marketing here at OmniTI too, I know the drill. All-in-all, I think the interview went rather well and fairly represents the benefits we've realized by deploying Solaris 10.

As many people already know, I'm a big fan of DTrace. Well, today I attended dtrace.conf(08) -- the first (un)conference revolving around planet DTrace. It was awesome. Many people who know me well have heard me say, "my good days are when I'm the dumbest person in the room." That's not to be confused with "I like having bad days." Instead, I like to be at my best and still struggle to keep up. Here at dtrace.conf(08), the people here are damn smart -- a bunch of higher level thinking.

I gave a demo of the PostgreSQL stuff we do using dtrace. I put back my SDT postgres probes into our internal packaging systems -- I hope that I can get those back into PostgreSQL soon... I talked very briefly about the ZFS magic we've experienced, but as the conference was focused intensely on DTrace, I saved most of that for PostgreSQL Conference East 08.

As usual, I particularly enjoyed Bryan Cantril's infectious excitement and pungent humor. I'd like to thank Sun and the DTrace Team for running the conference (as well as the sponsors). I really appreciate the effort. DTrace has empowered OmniTI to make our customers more successful. Hands-down awesome technology.

I've said it before and I'll say it again. ZFS and DTrace are the most impressive operating system advances I've seen this decade.