Esoteric Curio - OpenSolaris

Reconnoiter and another platform

nospam@example.com (Theo Schlossnagle) — Fri, 27 Jun 2008 17:27:27 -0400

Reconnoiter is coming along. Unlike most open source project, I tend not to talk about mine until their are really useful to people. Over the last year, I've adopted the unhealthy attitude that useful means "shiny front-end." So, I'm blogging to break that attitude and talk a bit about project that doesn't have a shiny front-end... yet.

Reconnoiter is built out of years of frustration using tools like RRDTOOL, Munin, Cacti, ZenOSS, Nagios, etc. etc. I have a lot of problems with these tools. First, they are not efficient. I need a powerful machine to monitor a mere 10k services. And it actually gets to be an engineering challenge to monitor 100k services with these tools. Also, the graphs are about 10 years old with respect to design and usability. I want something new, something fresh, and something that doesn't need a damn web UI to configure. Several people have asked, why are you reinventing the wheel? Why don't you just improve an existing product? My answer is that I want a well-thought-out product foundation so that I can trust all the bits. I want reponsibilities decoupled at the right spots. I want data in a form that the world can query and run reports the likes of which I have not concieved. I don't want the load on my monitoring machines to be 8. I want my monitoring system to check services and metrics when it planned to, not several minutes (or event 2 seconds) after it told me it would. Simply put, I expect it to work well, all the time. And, of course, I want it to work how I would expect it to work.

Reconnoiter was born out of the need to monitor the internals of many disconnected data centers with between 10 and 1000 machines in each facility. Monitoring can mean a lot of things, here I consider it to be the collection of metrics and awareness of their availability. In and of itself, monitoring is pretty useless, but it is the foundation for two critical pursuits in Internet infrastructure and business management: fault detection and trending.

Fault detection is as simple as understanding when something has faulted. However, knowing something is broken is easier than knowing something is about to break. Is it better to know that your machine just crashed because the chip slagged to the motherboard, or that the temperatures in rack 043 are rising unexpectedly? Answer: both, but I hope I only learn the latter and not the former. Truly, there are too many things to monitor... hundred or thousands of metrics on each piece of equipment. I can't reasonable go in and configure good/bad thresholds on each one. I want anomaly detection. I want a system that I can say: "this looks right, tell me when it stops looking right." That, to me, is a much need companion to tradition fault detection.

To me, trending is much more than drawing graphs... it is about intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you fucked up getting where you are now -- in the vain hope that you learn from your mistakes and plan better next time.

Reconnoiter is an attempt to build these things. Building a system requires starting with pain (need), solid structure and plumbing (good engineering). So, reconnoiter is underway. And this post is in mid-step:

It started on OpenBSD, and added support for FreeBSD, Mac OS X, Linux.

As of changeset [292], we have Solaris/OpenSolaris support.

We have a pretty nice front-end for trending under construction, but it isn't there yet. We'll have numeric data combined with textual "event" data on the same graphs. All that convenient stuff. Here's the rather plain-Jane graph you get now (because some people won't even read a post if it doesn't have a pretty graph):

Honestly, I don't know what the value of this post is, but people around here keep telling me that people should be aware of an open-source tool like this, even if it isn't finished (read: usable) yet. I say it isn't usable yet, but on our development instances here, we monitor 2892 production metrics across two data centers and the load never peaks past 0.10. I'm pretty excited about where this is going. Honestly, my favorite part right now is that I can configure and control the noitd checking nodes via a telnet console and it acts as if it is a piece of network equipment rather than an "application" -- as it should be IMHO.

ZFS. Respect.

nospam@example.com (Theo Schlossnagle) — Tue, 22 Apr 2008 23:16:32 -0400

Today someone asked me: "You speak about ZFS a lot. I know other people that talk about the latest filesystems with praise, but generally speaking they just don't have much to offer. Is ZFS that different?"

My answer is "yes." But, of course, I can't leave it at that. I'm not going to make a performance argument -- ZFS is fast in some cases and slow in others -- just like everything else. I think one of the things we've seen in the last 10 years is that everyone felt the need to come out with their own filesystem -- at least on Linux. So, you have to as yourself why. My personal opinion is that filesystems on Linux suck.

Most filesystems on the market support snapshots. No open source filesystems on Linux (that I'm aware of) support snapshots. Of course, you can use LVM to do block-level snapshots. First off, that's a pain in the ass w.r.t. storage provisioning. Other systems make the process of allocating and managing snapshots "not my problem." (simple and easy). Let's be frank, ext2 and ext3 are nothing to write home about. reiserfs, xfs, jfs, the list goes on and on.

There are a few closed-source filesystems that are really nice. Specifically Veritas Filesystem (VxFS) and its excellent layered volume manager VxVM which appears to have heavily inspired geom on FreeBSD. DEC thought it was so cool that they pulled it white-label into Tru64. Respect.

So, what makes ZFS so different? ZFS is a disruptive technology as it abolishes the sacred line in the sand between block devices, volume management and filesystems. This means it just make storage management easy. When I say easy... I mean easy.

So you want more space? Add more disks. Want to move from from failing disks to replacements? Tell zfs to add the new ones and tell it to remove the old ones. Read that report by Google about disk errors? ZFS checksums all data. My personal experience says checksums are good. Snapshots? Sure snapshot to your heart's content. We snapshot some systems hourly and never ever delete the old ones. Snapshots are really cool, but what if you could rollback to a snapshot? zfs rollback. What if you wanted to make a read/write copy of the fileystem or an old snapshot? zfs clone. You want to store a lot of raw data? zfs has built-in compression. Oh, and it is open-source.

Simply put. ZFS. Respect.

PostgreSQL: Looking under the hood with Solaris

nospam@example.com (Theo Schlossnagle) — Sun, 30 Mar 2008 09:21:05 -0400

For those interested, here is my slide stack from PostgreSQL Conference East '08. I think the title of the talk was "PostgreSQL: Looking under the hood with Solaris."

The presentation was 90 minutes long and had lots of shell-based show-and-tell. Obviously that stuff isn't available in the slides. I think it went over quite well. The audience was small, but hopefully people took away the a lasting impression of what DTrace has to offer and at least one person had the response: "By the power of Greyskull." Regardless, enjoy the slides!

Talking w/ Sun

nospam@example.com (Theo Schlossnagle) — Sun, 16 Mar 2008 23:21:55 -0400

New acquaintances walk away from their first conversation with me and either think that I am in love with a particular vendor or technology or they think I truly hate all technology. Both are true in some fashion.

The fact that I have an OpenSolaris feed on my blog might indicate that I'm a fan of Sun. The truth is I am and I am not. As is true of any large organization, it's really tough to be enamored with all of it. I am a huge fan of Solaris 10 and Sun's initiative to support strict ABI compatibility for stable interfaces, and I'm downright giddy about their ZFS and DTrace technologies. I think Zones/Containers are cool and I think their engineering team has some brilliant shining stars and is on the whole smarter than average. Yet, the OpenSolaris community is challenged in a lot of ways due to the corporate involvement by Sun that leaves me with a funny taste in my mouth. I'm luke-warm about Java and feel like their hardware initiative is going down-hill with bad quality problems on some of their new offerings compared to their spectacularly rock-solid history (sans the E4500).

Recently, I did an interview with Mark Thacker of Sun about our use of Solaris for their Solaris Podcast series. We've had some bad experiences here and there, but all-in-all it has been a win. DTrace has been a god-send and ZFS has saved my bacon several times. Anyone who's talked to me knows that I'm brutally honest and appreciate those that return the favor. I'll look at solution and I tell you what you did wrong. I don't tell you what you did correctly... after all it was all supposed to be done correctly. So, upon listening to this Sun podcast, several of my colleagues said: "that had to have been edited." As marketing people usually do, they attempt to limit the negative exposure as much as possible -- most notably, they removed a section about lack of tight integration between ZFS and Zones which has made for some very painful upgrade paths. We have marketing here at OmniTI too, I know the drill. All-in-all, I think the interview went rather well and fairly represents the benefits we've realized by deploying Solaris 10.

dtrace.conf(08)

nospam@example.com (Theo Schlossnagle) — Fri, 14 Mar 2008 21:28:49 -0400

As many people already know, I'm a big fan of DTrace. Well, today I attended dtrace.conf(08) -- the first (un)conference revolving around planet DTrace. It was awesome. Many people who know me well have heard me say, "my good days are when I'm the dumbest person in the room." That's not to be confused with "I like having bad days." Instead, I like to be at my best and still struggle to keep up. Here at dtrace.conf(08), the people here are damn smart -- a bunch of higher level thinking.

I gave a demo of the PostgreSQL stuff we do using dtrace. I put back my SDT postgres probes into our internal packaging systems -- I hope that I can get those back into PostgreSQL soon... I talked very briefly about the ZFS magic we've experienced, but as the conference was focused intensely on DTrace, I saved most of that for PostgreSQL Conference East 08.

As usual, I particularly enjoyed Bryan Cantril's infectious excitement and pungent humor. I'd like to thank Sun and the DTrace Team for running the conference (as well as the sponsors). I really appreciate the effort. DTrace has empowered OmniTI to make our customers more successful. Hands-down awesome technology.

I've said it before and I'll say it again. ZFS and DTrace are the most impressive operating system advances I've seen this decade.

Spine-tingling dtrace euphoria: "What the SCSI?"

nospam@example.com (Theo Schlossnagle) — Thu, 20 Dec 2007 08:52:01 -0500

I am oft in the position where SCSI is not working right for me. Whereas I always seem to be in the position where SATA over promises and under delivers. When SCSI misbehaves, it usually really matters. Where's my disk go? Time outs? And the king of all frustrations: wassup with that tape robot?!

This clearly puts me in the sysadmin geek category, but I had to share. I had a spin-tingling euphoria when I read the details of SCSI packet sniffer written in dtrace. OMG this is cool: http://blogs.sun.com/chrisg/resource/scsi_d/scsi.d-1.12.

Zimbra on ZFS and Zones.

nospam@example.com (Theo Schlossnagle) — Mon, 29 Oct 2007 22:13:59 -0400

So, we're moving from an old Cyrus installation to Zimbra. We considered moving to Dovecot, but the need for tight corporate calendaring that isn't hosted by big brother (Google or Y!) was too strong.

So, I call up Zimbra...

Me: Gimme that for Solaris.
Zimbra: No.
Me: Umm... It's open source and the "closed" bits are non-native Java.
Zimbra: Umm... It doesn't work on Solaris, just install Linux.
Me: Just give me a sound off-site backup strategy that works as well as ZFS.
Zimbra: Umm... Install Linux, it's easy.
Me: You're kinda missing the point.

I then turn to Sergey, master of brute force, on our SA team and said: you've got five days to run this on Solaris. Sergey says: "done."

From here, we test quite a bit, doing migrations from the existing solution and installation tests. Whiz, bang, whirrrrr. We're happy, and I go back to Zimbra with some trepidation.

Me: How much?
Zimbra: Fill out this form and fax it.
Me: Okay... [faxed form]
Me: I sent it.
Zimbra: Got it, you should have your license.
Me: Wow, that was easy.

While I like free software a lot. Boy is it refreshing to find a company that knows how to take your money without pain and suffering, like some companies I know.

I'm sure that Zimbra is just trying to limit the support platforms to help control their engineering costs. The support for Linux is a no-brainer -- it's the most popular new deployment platform for just about everyone. But, support for Mac OS X... as if there is demand for that in the enterprise?! WTF? I know all about developing mail products and I understand that Mac OS X support is often iintroduced due to the engineering staff owning MacBook Pros. But, come on... Tier 1 support for Mac OS X before Solaris, not very good market research here fellas.

Anyway, we've got Zimbra cooking on a sweet Solaris box attached to a Sun StorEdge T3. One zone runs Zimbra Network Edition and another zone runs Zimbra Open Source Edition. Both of these running on some nice ZFS mount points that are conveniently backed up with Zetaback... Bliss.

Oh yeah. One detractor to my bliss: I spent 2 days deleting old mail folders and eliminating cruft (built up over the last 10 years). 11 hours of imapsync to get my now down to 14 gigabyte mail store over is just painful. There should be a universal mail store format so that I can "burn" by old mail into a universal archive format and just plug that into mail any mail server. Right.

PostgreSQL warm standby on ZFS crack

nospam@example.com (Theo Schlossnagle) — Mon, 02 Jul 2007 21:06:32 -0400

So, the state of open source database replication is pretty sad. MySQL replication just doesn't cut it in many serious environments because the slaves can't keep up with the write load on the master. So, PostgreSQL right? Well, not so fast. PostgreSQL replication is handled in one of two ways: Slony or PITR (point-in-time recovery).

Slony provides all the same features as MySQL's replication (except that it is much harder to setup and maintain), but also boasts the same acute performance issues -- a busy master can easily outpace slaves, leaving them in the dust. Query-log-based replication is pretty flawed and while creative people will do whack shit to try to push the envelope this still doesn't make it a good method.

PITR is much more like Oracle's replication mechanism. PITR takes the WAL (write-ahead logs) and ships them over to the slave to be reapplied. This leaves you with an identical database (block for block) and a weak machine can easily keep up with a beefy master. In Oracle terminology "WALs" are called "archive logs."

So, with PITR, all our problems are solved, right? No. When using a PITR-style warm standby configuration the database is in "recovery mode" all the time. This means the database is "sorta up" waiting for the next WAL log to appear so that it can play forward through the transactions and "catch up" to the master: "continuous recovery." This means the database isn't available for queries. Now, Oracle works the same way. While Oracle is recovering, you can't use the database. However, using Oracle you can cancel recover, mount the database read-only, do some queries, unmount and begin recovery again picking right up where you left off. In PostgreSQL you cannot open the database in read-only mode and then later continue recovery as the act of opening the database (even in read-only mode) will deviate from the path of the recovery -- can we say design flaw?

While Oracle's "got game" on PostgreSQL, the concept of stopping recovery so that we can run queries on the slave isn't ideal. If my queries are substantial my "warm standby" will get colder and colder as it sits neglecting to apply archive logs. So, I want my warm standby and I want to be able to run long, heavy queries against it. And someone's going to give it to me!

However, I'm impatient. So, I'm going to make it work myself. Using the power of ZFS, I'm going to snap my PITR slave and clone it into a "disposable" "point-in-time" copy. This is really useful for running heavy reports.

The basic concept is this:

I've got PostgreSQL running on my box as a PITR slave. The master is pushing WAL logs over and this box is running in recovery mode.

Per best practices, my postgres data directory, xlogs and WAL archives are on different filesystems (ZFS of course).

intmirror/postgres/82_xlogs  64.1M  66.8G  64.1M  /data/postgres/82_xlogs
store2/postgres/82    10.5G  1.97T  8.15G  /data/postgres/82
store2/postgres/82_walarchives 14.4G  1.97T  3.89G  /data/postgres/82_walarchives

zfs create store2/clonedb
I create zone called "clonedb" with a zonepath /zones/clonedb
I make 'store2/clonedb' subject to an 'add dataset' in the clonedb zone configuration.
I setup the zone to run postgres just as the globalzone does.

Now, all I have to do is get a copy of the PITR stuff into that zone. There are a few caveats: (1) due to postgres' design the copy must be read-write as it will be destructive even in read-only mode and (2) it will still be in recovery mode, so I'll need the last WAL archive so that it can "finish" recovery before I bring it online.

ZFS gives us cheap, fast read-write clones of filesystems:

In the global zone:
zfs snapshot store2/postgres/82@clonebase
zfs clone store2/postgres/82@clonebase store2/clonedb/82

In the clonedb zone:
zfs mount /store2/postgres/82
zfs set mountpoint=/data/postgres/82 store2/clonedb/82

I need to make sure I copy the latest WAL archive from the PITR slave into the pg_walarchives directory on my clonedb zone. Then I just startup my postgres instance in the zone and touch the /data/postgres/82/failover file (this file tells my recovery script to stop recovering and start up normally). Viola.

It might sound complicated, but we just run ./clonedb_startclone.sh and in about one minute we have a fully operational read-write database and the PITR slave is merrily continuing recovery.

# ./clonedb_startclone.sh
[Mon Jul  2 20:37:14 EDT 2007] Stopping postgres in clonedb
[Mon Jul  2 20:37:20 EDT 2007] Dropping clone and base snapshot
[Mon Jul  2 20:37:38 EDT 2007] Snapshot store2/postgres/82
[Mon Jul  2 20:37:39 EDT 2007] Clone to store2/clonedb/82
[Mon Jul  2 20:37:41 EDT 2007] Mount store2/clonedb/82 at /data/postgres/82 in clonedb
[Mon Jul  2 20:37:43 EDT 2007] Copy last WAL [0000000100000016000000FA]
[Mon Jul  2 20:37:43 EDT 2007] Make it active [induce failover]
[Mon Jul  2 20:37:43 EDT 2007] Start postgres in clonedb
[Mon Jul  2 20:38:07 EDT 2007] System up

Now I can run long data mining queries and other complicated reports against my standby database. No load is induced on the master database at all (so no concern about negative production service effects) and the standby recovery is continuing on unaffected, so from the failover point-of-view nothing has changed either. I am not even limited to one zone! Any time I'd like to, I can just "snap" myself a new query slave. It's a cheap, mutable, entirely disposable copy. Nice.

It's worth noting that this same technique should work like a charm on Oracle as well. Also, it should work well with any other filesystem that supports copy-on-write cloning -- though I don't know of any other than ZFS.

This, in a long line of things, just lets you know that when your database doesn't quite have the spunk to finish the race, today's operating systems are actually powerful enough to drag them across the finish line.

ZFS rules: congratulations to FreeBSD, Sun and Pawel Dawidek

nospam@example.com (Theo Schlossnagle) — Fri, 06 Apr 2007 09:29:15 -0400

I am very pleased to learn that shortly (a year or so) after Sun's opening of Solaris under the CDDL open source license that communities outside of the Sun/Solaris community are truly seeing benefit. Pawel Dawidek announced yesterday that ZFS is now in FreeBSD HEAD and will be available in 7.0. This is very exciting news. We use ZFS and many of its advanced features extensively here at the $DAYJOB and I can say with authority that ZFS is damn cool juju.

We manage quite a few FreeBSD machines (in addition to Linux and Solaris). It's good to know that some of the elegant solutions we've deployed on Solaris leveraging ZFS will now translate well to another platform we work with.

Sun/OmniTI PostgreSQL Case Study

nospam@example.com (Theo Schlossnagle) — Mon, 19 Feb 2007 23:36:45 -0500

I've blogged a few times in the past about running large PostgreSQL applications on Solaris. I've also spoken about the same adventure at various industry conferences. I'm pretty excited to see that the case study that we worked on with Sun has finally hit the bit pipes of the Internet.

I'll note that we run Solaris, FreeBSD, OpenBSD, Mac OS X, and Linux here at the $DAYJOB. A whole lot of Linux, actually. The reason we run these various things is that they are all UNIX or UNIX-like and all make us happy. Each is a different, yet effective, lance to shatter the windmills at which we tilt. Each OS has its issues, none is perfect.

This particular tale is one of a datawarehouse application and astoudingly effective application of Solaris 10 and PostgreSQL 8 to boast. Enjoy.

OmniTI’s move from a proprietary application to PostgreSQL for its customer was prompted by growing database requirements that were threatening to send software costs skyrocketing. The company’s existing online transaction processing (OLTP) and data warehouse capabilities strained to keep up with the demands of a half-terabyte OLTP database peaking at 10,000 transactions per second and a data warehouse database consuming 1.2 terabytes.

ZFS send^H^H^H^H trickle.

nospam@example.com (Theo Schlossnagle) — Wed, 27 Dec 2006 15:13:04 -0500

One of the things that ZFS boast most is its scalability -- Z is for zetabyte after all. Trivia question: what is the first thing you do after you put data on your production ZFS volume? That's right, you back it up to your backup infrastructure. A lot of systems use tar or other archive like derivatives to manage backups. This technique is particularly awful with databases. Databases usually consist of very very large files (multi-gigabyte) that have minimal changes to them. With full archive systems, any attempt at incremental backups results in horrible space and time inefficiency as a small (8192 byte) change in a datafile will necessitate the whole file to be backed up in the next incremental.

Enter block-level incremental (BLI) backups. The idea here is that you ask your filesystem to track which blocks change from a certain moment in time. And you can ask the filesystem for all blocks of a filesystem (view consistent, of course) and then later ask it for the changeset. In other words its like doing:

snapshot FS as 'base', backup 'base'.
wait a day
snapshot FS as 'inc1', backup diff 'base' 'inc1'

Filesystems have supported this type of behavior for a while now (Veritas VxFS has a magnificent implementation). Needless to say I was ecstatic when I read the zfs manpage and learned of the 'zfs send' and 'zfs recv' operation. Functionally, they implement BLIs.

We have a database on which we have around 1TB of information on zfs. So, I figured we'd whip together a script to tie in zfs send (including incremental support) to our Veritas NetBackup infrastructure.

We have three mount points that we need to snap and send to NetBackup, so I create three FIFOs on disk and fork off three parallel 'zfs send' operations. Then I fork off three parallel netbackup jobs (one for each FIFO). We have three tape heads so, they all actually run in parallel and should fly like the wind (all over GigE).

# date; ./zbackup.sh -s -l 2006121402; date
Thu Dec 14 12:58:43 EST 2006
./zbackup.sh:
  backuplabel: 2006121402
  full

zfs destroy intmirror/xlogs@lastfull
zfs destroy xsr_slow_1/pgdata@lastfull
zfs destroy xsr_slow_2/pgdata@lastfull

Backing up as '2006121402'
starting postgres backup on label 2006121402
zfs snapshot intmirror/xlogs@lastfull
zfs snapshot xsr_slow_1/pgdata@lastfull
zfs snapshot xsr_slow_2/pgdata@lastfull
stopping postgres backup on label 2006121402
/sbin/zfs send intmirror/xlogs@lastfull >> /pgods/scratch/intmirror:xlogs.lastfull.full &
/sbin/zfs send xsr_slow_1/pgdata@lastfull >> /pgods/scratch/xsr_slow_1:pgdata.lastfull.full &
/sbin/zfs send xsr_slow_2/pgdata@lastfull >> /pgods/scratch/xsr_slow_2:pgdata.lastfull.full &

Sat Dec 23 15:39:47 EST 2006

SWEET JESUS! That's a 9 day, 2 hour, 41 minutes and 4 second backup. Somehow I think that doing daily incremental backups is out of the question. I tried zfs send redirected to /dev/null (just to demonstrate that netbackup was not the bottleneck) and, as expected, there was no noticeable speedup. I've tested this on some other machines and got the send operation to run quite fast. However, any time a very competitive I/O load is added, it just suffers miserably and becomes so slow that it is useless.

Reading the source code to the ZFS layer leads me to believe that all the operations for doing the send are scheduled serially (each after the previous completes) and compete equally for system I/O with all other processes. I saw no intuitive way to make the ioctl()s with ZFS act as if they were more important that other things going on in the system. This leads me to believe that it may not be so easy to fix. However, those Sun engineers have wicked tricks up their sleeves and tend to pull of some amazing feats. So, here's hoping!

Until then, I hereby suggest that the 'zfs send' be renamed 'zfs trickle'.

Choosing Solaris 10 over Linux

nospam@example.com (Theo Schlossnagle) — Tue, 28 Nov 2006 22:31:17 -0500

The Domain

So, at the $DAYJOB, we were faced with building a large operational data store. Large has many meanings to many people. I've written about this before, but I'll reiterate the scope: (> 1TB data, thousands of tables, several tables with around one billion rows). So, for a variety of reasons, we chose PostgreSQL. I've written about that choice a few times, but didn't write about the choice to use Solaris.

A Bad Choice

So, I'll start by saying we chose Linux -- CentOS 4 (an RHEL4 clone). The box we chose was a 4 processor Opteron with 16GB of RAM, three internal drives and the rest is fiber-attached storage. While I like XFS a lot as a file system, but I've seen some odd issues here and there (specifically on our 1TB mail server here and out 1TB subversion server). So, just to be safe, we used the tried and true ext3.

It started out well. One day /data/ (our large postgresql data mount point) suddenly went read-only. Postgres, of course, was quite displeased with this occurrence. So, naturally, I tried to fix the problem by umounting and mounting again, all to no avail. It turns out that a reboot was required to rectify the issue. While this was disturbing, we rebooted and continued on with life. The more annoying issue was the subsequent 18 times this occurred. The show stopper was the 20th time it failed; upon reboot I found catastrophic data loss.

To set the scene appropriately: we run Linux in a lot of places and run FreeBSD and OpenBSD and Solaris. Where we run these it runs well -- because when it doesn't we replace it with something does. I love FreeBSD, but I have some application code that will make it kernel panic inside 5 minutes. Long story short, fear PostgreSQL on 64-bit Linux. Once bitten, twice shy... 20 times you're an idiot.

A Better Choice

So, we chose Solaris 10. Why? We've run a lot on Solaris and been quite pleased with its stability. It has excellent support for enterprise storage hardware and multi-processor AMD Opteron systems. We've used VxFS (Veritas File System) and it has been a "good life." Solaris 10 sports a new file systems called ZFS which boasts a lot of the features of the VxFS file system (but not quite the performance). ZFS's volume management, built in compression, snapshot capabilities, and simple management makes it a hands-down winner over LVM (Linux Logical Volume Manager) and ext3. Now we have two 2.7TB ZFS pools, soon to add another 1.4TB pool, and management has been quite painless.

Another issue that we had with PostgreSQL (coming from Oracle) was severe inability to introspect the database. Why is a query slow? How many disk reads did it do? Which locks did it acquire, when and how long did it block waiting for the lock to be granted? If a query hits disk, which spindles were accessed? PostgreSQL simply does not provide an interface to this information. How, you ask, does Solaris help with this? Enter DTrace.

DTrace allows us to dynamically instrument the undercarriage of PostgreSQL (see what it is doing from the application level all the way down through the kernel). Because you can instrument application space right along side kernel space, we can see the queries being performed, the SQL plan being executed, the memory allocations, lock acquisitions, disk reads and writes, simply put: everything. Using DTrace we were able to develop a PostgreSQL administration toolkit that provides information such as the number of blocks read and written to each individual spindle over the course of a given query.

We also use SMF, but I honestly don't think that buys us anything really special. It's nice that it "just works," but quite frankly you can my SysV scripts "just work" as well.

In the works is a set of PostgreSQL administrator tools that transparent database-level access to systemic information about PostgreSQL back end processes.

Not everything is peaches and cream.

One of the nice features of ZFS is the snapshot backups and the ability to "dump" the differences between one snapshot and the next. This, in essence, is a BLI: block-level incremental backup. ZFS manages this on a "zobject" level as I understand it, but it should accomplish the same thing. It means that if you have only 1GB of changes on a 5TB filesystem, it is feasible to restore a 5TB base image and then a 1GB changeset to get a block-level accurate restore. This in turn means that you can backup 5TB once per week an then each day backup only the changes between that "level 0" backup and the current snapshot (which should be small and manageable). I love this feature -- but hate that it doesn't work. On our system, at least, a "zfs send" of the differences between one snapshot and the next can take 40-50 hours. This appears to be a flaw in the ZFS send works in that it doesn't prioritize its I/O requests over normal traffic. I hope the Sun guys get to work on this one soon!

Review

After running the system for some time now, I can say that Solaris 10 was an excellent decision. We've since launched four more respectably sized PostgreSQL instances on Solaris 10 with equal success.

PostgreSQL performance through the eyes of DTrace

nospam@example.com (Theo Schlossnagle) — Fri, 27 Oct 2006 14:17:18 -0400

We've been doing a lot of PostgreSQL work lately and we have one largish system (terabytes) that runs on top of Apple XServe RAIDs. While people argue that SATA is getting better, let it be understood that Fibre channel SCSI drives rule. The difference between carrier class storage and "enterprise" (a.k.a. commodity) is pretty tremendous.

While this system will eventually make good use of the XServer RAIDs and long-term storage containers for write-once read-many data tables (archives), the "fast" storage it currently tied up and we've been "making due" with all of our heavy read/write activity on 14 SATA drives. It is always advisable to understand the access patterns of your database so that you can put files that are heavily utilized at the same time on different spindles. This allows for a set of drives heads to be occupied with one style of work load without being distracted by another. In big databases, getting more spindles is a must and usually not a budgetary concern. BUT, how do you know which files?

Oracle provides enough information inside the database itself for a DBA to do an assessment of which datafiles could benefit from running on separate spindles. PostgreSQL's statistics exist, but are very lacking in comparison. The rule of thumb is that you pur indexes and data on different spindles. In big systems you don't follow "rules of thumb" you instead use them as the basis for experimentation and emperical analysis. Big systems often vary widely in their configuration and stress points and the "rules of thumb" are for generic problems, while they may hold true on your system, "don't jump."

So, what is a PostgreSQL user to do? First, run Solaris 10. Second leverage Dtrace to really answer the age old question "where does it hurt?" I scratched together a little perl/dtrace to measure the min/avg/max read() and write() times induced by PostgreSQL (yes it uses read() and write() for all of its I/O ops on UNIX) and break it down by filename. I initially used the io::: provider to only measure the ones that induced physical reads against media (didn't come out of buffer cache). While it is really cool that you can do this, we would miss the reads that would happen we moved things around and blew our buffer cache -- so I opted for all reads (less cool : more useful).

After we collect metrics from all the file I/O, we connect to PostreSQL and ask it which database objects correspond to the files we see being read and written to. The result is a simple prstat-like output by database object of the number of reads/writes performed along with the minimum, avergage and maximum turn-around time for the operation in milliseconds. Quite useful for understanding "what hurts."

So the command 'pg_file_stress -d dbname -u postgres -i 30 -n 20' would yield useful output like:

           FILENAME/DBOBJECT                                      READS                 WRITES      
                                                         #   min   avg   max     #   min   avg   max
ods_users                                             9494    84   144  1039     6     0    34   207
ods_tblfl_unsub_users_p_2006_10                          1   201   201   201   127     0     0     0
ods_tblusersanswers_p2006_12_ix_timestamp                1   179   179   179     0     0     0     0
tbluser_address                                        307    91    91  1076     0     0     0     0
pg_rewrite                                               8     0    93   749    12     0     4    59
ods_tblusersanswers_p2006_11_ix_timestamp                1    80    80    80     0     0     0     0
ods_tblhits_sum01_p2005_11_partner                      20     0    60   897     0     0     0     0
pg_depend_reference_index                               38     0    46   700    48     0     0     0
ods_tblfl_unsub_users_p_2006_09                          1    28    28    28    94     0     0     0
pg_depend_depender_index                                29     0    22   407    11     0     1    14
ods_users_pk                                           306     0    16   777     0     0     0     0
ods_tblhits_sum01_p2005_10_partner                       4     0    12    48     0     0     0     0
pg_statistic                                           158     0     9   819    88     0     0     0
ods_tblhits_sum01_p2005_11                            5279     0     4  1102     0     0     0     0
ods_tblhits_sum01_p2005_10                            1277     0     3   755     0     0     0     0
ods_tblusersanswers                                   4295     0     1  1422     0     0     0     0
pg_statistic_relid_att_index                            41     0     1    67    31     0     0     0
pg:3083076                                            1702     0     1  1015   808     0     0     0
mv_users                                              2751     0     1  1036    26     0     0     0
users_tx_p20061022                                    2264     0     0   667     0     0     0     0

9494 read operations with an average turn around time of 144ms. Sweet Jesus! I suppose that could use its own spindles. 307 ops at 91ms ain't pretty either.

More than demonstrating how to solve our problem, this demonstrates the acute need to decommission the box using our fast storage and swing those units over to our database of pain.

ZFS and mixed results

nospam@example.com (Theo Schlossnagle) — Sat, 26 Aug 2006 20:58:45 -0400

We've had great luck with ZFS on some "largish" systems. I haven't worked out how to blog about them yet as they contain a lot of client specifics, but I now have ZFS in a happy place of my mind. However, most of the stuff we do doesn'ts go over NFS. So, Eric (who leads out operations group) was tasked to solve some heterogenous home dir issues across a large build cluster we have at the $DAYJOB. His experience with NFS on ZFS wasn't so golden as mine. It didn't turn out all bad, but it's really good to know the pitfalls before you start a project -- so read up.

Big Bad PostgreSQL

nospam@example.com (Theo Schlossnagle) — Wed, 26 Jul 2006 17:21:36 -0400

I just gave a talk at OSCON on PostgreSQL. Specifically, it is a talk on migrating a portion of a large data architecture from Oracle to PostgreSQL. It is an honest assessment of what was gained and the pains involved. Several people have requested the talk be put online. Here it is: