Esoteric Curio

Web: you can now not suck.

nospam@example.com (Theo Schlossnagle) — Thu, 25 Jun 2009 23:33:33 -0400

In perhaps a new trend, I’m blogging from 39011 feet (or so says the seatback in front of me). I’m traveling back home to the east coast from San Jose, CA where I attended (and spoke) at this year’s O’Reilly Velocity Conference.

I participated (and blogged) about the Velocity Summit in which I’ve participated for the past two years. The summit is the unconference preceding the real conference that help the organizers digest current hot topics and better define the conference track for the actual conference. The summit itself is filled with enough brain power to warp space-time, so I drop everything to go to that.

Ironically, despite being a well respected authority in web site (and general internet) scalability and performance, my talk proposals for Velocity 2008 were not accepted — I clearly need to write better proposals. This year, I managed to work my way into the workshop track on Monday. Despite having a bad headache and feeling "off" the day before, I managed to get my act together and put on an A-game for my workshop. For those of you interested, here is my scalable09 slide stack.

I thought I’d take a moment to talk about what I liked about the conference and what I think could use some improvement. I realize this is a down economy and that might be a legitimate justification for some the actions that resulted in some of my disappointment.

First, the negative. I usually start with positive and end with negative because I’m a pessimist. However, all in all the conference was awesome, so I thought I’d get my short list of gripes out of the way early.

O’Reilly is infamous for throwing good conferences for geeks. In my opinion, the field of web operations has been so severely neglected and applies so broadly to the world today that this conference needs to be for everyone.

In the next conference, I’d love to see a technical business track. Several of the talks I went to spoke to the dollars and cents lost or earned by paying the right amount of attention to web site performance and better operational paradigms. I thought a lot of the topics would be very useful to business managers.
The first day was not video taped and the second and third day were only half video taped. Come on guys, ante up. The attendance fee was substantial, you can afford to give your attendees the value of watching what they had to choose not to attend. I like the option when I go to a conference to choose a session that seems interesting so that I have the opportunity to participate, but often times I find that another session was top notch and I expect to be able to later review a recording of that.
Lastly, and this is the most significant. While I thought the conference was extremely well executed (excellent job Jesse, Steve, all your support, and most definitely O’Reilly), it lacked sufficient PR and marketing outreach. I talked with several journalists (as a part of my normal day job) while I was at that conference and not one of them was aware of Velocity — simply embarrassing. Given that the Structure conference was in town that same week, O’Reilly should have invested more in their PR and marketing outreach. It would have resulted in a substantially increased audience and a better venue for teaching the world to run a faster web.

Now that I’ve griped and aired my disappointment. I can focus on the gobs of awesomeness that was Velocity.

The conference was put on quite well from an operational perspective. Things started on time, A/V problems were non-existent. Like an idiot, I managed to lose my MacBook Air power adapter and the A/V crew managed to recover it for me. Conferences just plain suck when they have technical difficulties; this one had none.
The two tracks at the conference were extremely well articulated and while I wanted to be in both all the time (as OmniTI is a full-stack company, we care about both equally) it was an excellent split.
One track was performance which focused intently on user-perceived performance. This was largely front-end (HTML,CSS,JS,etc.) but also had a healthy amount of deep stack performance discussion as well including the often ignored, but much deserving networking aspect of delivering the web to users.
The second track was operations and I feared it would be bunch of blind “The Cloud Solves All Our Problems” sessions. Much to the contrary, it focused heavily on operational strategy and and what it takes to execute tactically. There was a bit of cloud here and there (okay everywhere), but there was very little blind and ignorant mentality that launching in the cloud was a solution to a hard scaling problem.
I had the opportunity to see a lot of old faces, but spent most of my time meeting new ones. The opportunity to learn more about other people’s problems is what completes me as an engineer. The people I met at this conference were both honest and open and provided a fabulous and refreshing perspective on what today’s performance and scalability problems really are.

In my workshop, I spent about 20% of the time discussing the philosophy of being a good engineer and 80% discussing practice (non-cookbook) with examples and advice. The basic message is that systems are complex and you must think of all the parts holistically or its a recipe for disaster — or failure.

Two of my favorite talks were Nicole Sullivan's “The Fast and the Fabulous: 9 ways engineering and design come together to make your site slow” and “ 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” by John Allspaw and Paul Hammond. while Nicole's presentation, like mine, was not recorded, the other was and if you want to break down the divide between operations and development, it is a must see.

All in all, I would encourage everyone reading this to attend next year's Velocity conference. I am certain you will walk away with knowledge that is both valuable and applicable.

What time is it?

nospam@example.com (Theo Schlossnagle) — Tue, 16 Jun 2009 15:28:40 -0400

PostgreSQL has pretty awesome date/time functionality. I've used a lot of database and the functionality and thoroughness of the treatment of dates and times (and particularly timezones) is unparalleled. As much as I'm impressed with it, I knew there would come a time where the outcome of all that cleverness would backfire.

Recently, I was doing some data partitioning. I split a couple of largish (approximately billion row) tables up into month segments. I wrote a tiny little pl/pgsql function that takes a parent table, and creates an inherited child with the right indexes, triggers, check constraints (for constraint exclusion) and permissions. I renamed the big table something transient, created a new parent table with the old table's name, made the old unwieldy table a child of that table and then created a whole bunch of new partitions. This allowed me to pretty much ignore my shenanigans from the application side. Once I created the partitions, I need to back populate them. To do this, I did the following:

ALTER TABLE newchildtable_200903 DISABLE TRIGGER ALL;
INSERT INTO newchildtable_200903
  SELECT * FROM oldcrappytable
  WHERE whence >= '2009-03-01 00:00:00-00'::timestamptz
      AND whence < '2009-03-01 00:00:00-00'::timstamptz + '1 month'::interval;
DELETE FROM oldcrappytable
  WHERE whence >= '2009-03-01 00:00:00-00'::timestamptz
      AND whence < '2009-03-01 00:00:00-00'::timstamptz + '1 month'::interval;
ALTER TABLE newchildtable_200903 ENABLE TRIGGER ALL;

Suffice it to say, this did not do what I wanted at all.

PostgreSQL's interval type is one of its more clever features. The idea that a month isn't always equal to a month is a Gregorian truism. So, PostgreSQL is design to "do the right thing" and consider a month in the context of another argument. A month in the above example is a month with respect to March. Right? No.

The lacking part here is the timezone. I do the partitioning in UTC (because I'm not insane). So, I need the month of March in UTC. Although I explicitly stated '2009-03-01 00:00:00-00' in UTC, PostgreSQL interprets that in the client's timezone... and then adds a month. I'm in US/Eastern which is trailing UTC by four or five hours and thus the reference starting point is actually in February of 2009, which only had 28 days. So, the latter inequality up there does not do through the end of March!

I would argue that this behavior is invalid, because of the extremely unexpected results of the following simple test case:

postgres=# select ('2009-03-01 00:00:00-00'::timestamptz + '1 month'::interval);
2009-03-28 19:00:00-04

postgres=# set timezone = 'utc';
SET

postgres=# select ('2009-03-01 00:00:00-00'::timestamptz + '1 month'::interval);
2009-04-01 00:00:00+00

Here I get a completely different date/time if I ask what appears to be a very unambiguous question depending on whether I'm left or right of the Prime Meridian.

Cloud storage primer

nospam@example.com (Theo Schlossnagle) — Thu, 11 Jun 2009 16:41:27 -0400

I post here, I post there... I certainly don't post everywhere. Anyway, I wrote an article that discusses the basic technologies that power large cloud-like storage systems. Thought you might be interested.

Reconnoiter Update

nospam@example.com (Theo Schlossnagle) — Thu, 04 Jun 2009 08:04:34 -0400

A little over a year ago, I started in on a project that was of significant scope. Not a few scripts hacked together, nor a conglomeration of pre-existing tools, but rather a carefully engineered product. What product is this? Reconnoiter.

About 10 years ago, we were neck deep in large scale e-mail architecture. We felt pain, we were up at 3am every night attempting to make systems work. Finally, we decided enough was enough and started a skunkworks project to build a better e-mail server. Well, that turned out pretty well. It's got considerable momentum at this stage and is leading the industry as the most advanced digital messaging platform on the planet.

Over the past several (12) years, we have run operations for small and large sites alike. We're responsible for waking up and fixing things at 3am when they are not working. We're responsible for not only designing highly scalable architectures, we're responsible sticking around and seeing them through to the finish. Many people are writing tools for management; two in the recent spotlight are Puppet and Chef. We had very little pain in the arena of provisioning and maintaining systems. I have a theory as to why that is, but that is a topic for another monologue. One of the distinct pains we have suffered since we began revolves around monitoring.

The first issue is that monitoring is two things:

trend analysis for long-term capacity planning and postmortem analysis,
fault detection.

There are many tools today that are hard to use and fail to address our needs for managing thousands of very different machines. Worse, the tools do only one or the other. This means that we must invest time checking disk space in the fault detection tool to alert us when it is "too full" and configure a similar check in a trending tool to show us historical information. Some patchwork was introduced into fault detection tools like Nagios to add trending features... and when I use it, it is clear it was not central to the design.

I have a lot of gripes, but I won't go into all of them. Suffice to say I have them and I think they are the true fuel for developing a next-gen tool to make operations folk suffer less. Combine that with the combustive talent of the engineering group at OmniTI (and now a few outside it) and the oxygen that the open source community provides, and we'll be having a barbecue in no time.

So what's new?

A lot has happened in recent months on the Reconnoiter front. Here's a set of highlights:

OmniTI labs got its IANA enterprise number. On that we build SNMP trap support into noitd.
Reconnoiter got its IANA assigned port number, no more picking random ports. 43919 or bust!
We support durable streaming (for collection and storage of metrics)
We support temporal streaming and check cloning for real-time, high-granularity metric collection. This means cross-check (even cross-datacenter) real-time graphs for pretty awesome event correlation.
We integrated Esper as a streaming database (complex event processor) as the foundation on which to build fault detection. There's a long way to go here, but the plumbing is in place.
We build a custodial daemon capable of running nagios-type checks to ease adoption -- even though it made me cry.
We started sitting on #noit on freenode, though it is still eerily quiet in there.
We got some new users with the brilliant patience for alpha software and excellent feedback!
I'll be talking about Reconnoiter at the 2009 O'Reilly Open Source Convention.
It's starting to get some serious attention, so it's time we add some polish!

We've been slowly introducing our managed clients to Reconnoiter and we have, at this point, about a terabyte of metric data. In Reconnoiter, there is no default action to discard data. Yes, that's right. Go buy more disk. It's cheap. You'll thank me next time you have an anomaly today that you think reminds you of one seven months ago... and when you go look at the graph you actually find all the data at its original granularity.

Oh God, SMF rocks.

nospam@example.com (Theo Schlossnagle) — Tue, 26 May 2009 22:34:27 -0400

I just saw this. I'm happier than ever that I have this.

What is professional services?

nospam@example.com (Theo Schlossnagle) — Thu, 30 Apr 2009 16:14:24 -0400

I'm always humbled by the praise our company receives. Some go so far as to say we're brilliant, and while that's very nice to hear, brilliant is a very high bar to set. We are, I think, some of the smartest people you'll find that let you take all the credit. We think this humility is what it means to be a professional services company, and this selfless mantra is the key to what makes OmniTI the best web company you've never heard of.

Ben Laurie is one of my dear friends and colleagues in open source. He both commands and deserves a tremendous amount of respect. Ben introduced me to one of my favorite quotes:

"It is amazing how much you can accomplish when it doesn't matter who gets the credit."
-- Harry S. Truman.

When I first heard that, I liked it. After many years of stewing on it, I love it. It embodies what it means to be a truly valuable professional consultant. Sure enough, it also embodies what it means to be a good citizen. Forget the blame, forget the credit... focus on teamwork and bring progress and success.

As a professional services firm, it is your profession to service others. Servicing others isn't always glamorous, it isn't always exciting (though I find it remarkably stimulating), and it certainly doesn't put you directly in the limelight. The point of it all is to enable another to succeed. It is your goal, it is your mission and, if you want to be good, it has to be your passion too. The one thing that is often missing in all of this is credit.

Perhaps that is one thing that I've come to accept after many years in the pro-services industry. If you do your job well, you know that you are essential to the success of your clients and you can be proud of the work you do for them. You aren't in the limelight because, at the end of the day, your goal is to put your clients in the limelight.

Good clients however (and so many of OmniTI's clients are just spectacular) will always act as a reference and share how essential you were to their success. I've spent a long time building OmniTI and OmniTI is truly its people. It is the people that deliver the services, that help the clients, that solve the problems. These people are smart and dedicated. To me, completing the job and earning the respect of the colleague sitting next to me in the trenches is deeply satisfying.

That is a job well done.

Disk benchmarking with dd -- don't

nospam@example.com (Theo Schlossnagle) — Tue, 14 Apr 2009 22:50:14 -0400

I heard some rumors float around about using dd as a simple test for disk throughput. I'd like to verbosely say, "that's a bad idea."

I'm going to log into a crappy system with two extremely slow 1.5TB SATA drives in RAID 1. Yes, this is a production machine and it's goal in life is to store a lot of things and serve some of them infrequently -- as such, its configuration is well suited for that task.

; /bin/time sh -c "dd if=/dev/zero of=ddfile bs=8k count=2000000"; /bin/time sync
2000000+0 records in
2000000+0 records out

real       53.9
user        1.2
sys        28.4

real        0.2
user        0.0                  
sys         0.0

I'll note first that the second set of times is the sync and we see it was effectively free. Second, we got 304MB/s. In RAID 1 we have to write to both drives, so in the best case we get the performance of the worst spindle. 304Mb/s seems a tad high.

; /bin/time sh -c "dd if=/dev/zero of=ddfile2 bs=8k count=2000000"
2000000+0 records in
2000000+0 records out

; /bin/time dd if=ddfile of=/dev/null bs=8k
2000000+0 records in
2000000+0 records out

real       36.9
user        1.3
sys        35.5

The first statement blows any buffer cache we might have. Then we see a read of a 16GB file that sustains an average of 444 MB/s (over two spindles). That too seems a little high.

Oh wait, I had compression on. Let's rerun that with it turned off.

; /bin/time sh -c "dd if=/dev/zero of=ddfile bs=8k count=2000000" ; /bin/time sync
2000000+0 records in
2000000+0 records out

real     3:17.9
user        1.2
sys        29.5

real        0.2
user        0.0
sys         0.0

Interestingly, still the sync is dirt cheap (ZFS pretty aggressively writes back) and we're at about 83MB/s. That's the sweet sucking sound of mirrored SATA disks.

Now reading it back after blowing the ARC (ZFS's version of buffer cache):

; /bin/time sh -c "dd if=/dev/zero of=ddfile2 bs=8k count=2000000"
2000000+0 records in
2000000+0 records out

; /bin/time dd if=ddfile of=/dev/null bs=8k
2000000+0 records in
2000000+0 records out

real     2:09.3
user        1.2
sys        13.1

127MB/s, as expected we see a better, yet still crappy throughput from our drives on reading as we're coming from two spindles instead of one.

Long story short: modern filesystems can do whack stuff to your workloads. Use a comprehensive workload generator for I/O benchmarking. Preferably one that can simulate something resembling a real workload. Greg mentions bonnie++ in his post about benchmarking. Bonnie++ is a legitimate benchmarking tool, but for generating real workloads, I suggest filebench. It might be a bit more work, but at least you can use the results!

Nice write-up on zones.

nospam@example.com (Theo Schlossnagle) — Sun, 12 Apr 2009 10:31:29 -0400

My colleague Mark Harrison at OmniTI wrote a nice little piece on Solaris Containers (Zones) and how we use them (a lot). It's a short and simple read, good for psuedo-technical people that want to know a but more about Zones.

Robert, spreading the wealth as usual.

nospam@example.com (Theo Schlossnagle) — Thu, 02 Apr 2009 22:36:55 -0400

I'm pretty excited that Robert Treat is talking at PostgreSQL Conference East 2009. He'll be talking about the upcoming release of PostgreSQL 8.4. The postgres product is pretty complete and has been pretty complete for quite some time. I've been noted for saying I wouldn't use a database feature in production that hasn't been available (and tested) for several years. As such, it is a bit ironic for me to say I'm excited for a new release of a database product. Let me explain myself.

I'm excited by several of the new features coming in PostgreSQL 8.4 because the sooner they are released, the sooner I can start my count down to when I'd trust them in production. Okay, that's not entirely fair. When you engineer things to embrace failure it is easy to launch newly released features with less risk. However, we run people's online businesses. Some things (e.g. those that must comply with these specifications), even if engineered to allow for failures, you simply don't take the chance. For other things, you can embrace failure and improve the product through feedback by running them in a globally trafficked, high impact environment.

Whatever the circumstances, I'm excited by the pending PostgreSQL 8.4 release. Perhaps Robert will give the same presentation to OmniTI at one of our weekly development sessions so that we know what the hell is going on.

Make sure you check out Robert's talk:

Another year, another PostgreSQL release, and once again this release packed full of new features that will help you whether you develop apps inside or outside the database, or you just need to keep them running. This talk will give an overview of the new features available in 8.4, and give you pointers to talks during the rest of the conference you'll want to focus on to get the most out of 8.4.

Databases performing? Oh my!

nospam@example.com (Theo Schlossnagle) — Tue, 31 Mar 2009 15:22:35 -0400

So, there's this really neat little conference being run by Percona. It's in the Santa Clara Convention Center on April 22nd and 23rd. If you are in the bay area, you should come check it out. It's free and the speakers list is simply smashing.

I'll be giving a talk on a largish PostgreSQL install that happens to be on Solaris on ZFS. So, interesting stuff all around.

I don't make it out to California that often, so if you want to catch me -- that'd be a good time. See you there!

First BWOSUG Meeting

nospam@example.com (Theo Schlossnagle) — Tue, 03 Mar 2009 21:47:18 -0500

I'm excited to announce the first ever BWOSUG meeting. On March 11th we'll be meeting up for the first time to geek-out on OpenSolaris technologies. To break of from the dry tradition of blind zealotry, we're going to shake things up a bit:

I've found that there's no true appreciation for technology like pain and suffering. As such, we'll be telling war stories of bugs encountered, havoc caused and work arounds applied. This meeting should get quite a few people talking and hopefully be very educational.

Join us at the OmniTI offices at 6:30pm on March 11th, 2009!

OmniTI
7070 Samuel Morse Dr. Ste 150
Columbia, MD 21046

Fuck the blind.

nospam@example.com (Theo Schlossnagle) — Sun, 01 Mar 2009 16:08:45 -0500

One of my favorite jokes always results with a wry grin on my face and ends with the punchline: "fuck the blind." Like many jokes, it is not politically correct. The only reason these jokes are funny is that they expose prejudices and injustices that still exist. Many of us laugh at these things and find little bits of truth in ourselves. The best of us reflect on that and increase our level of tolerance for all things different. I, like most Americans, have a long way to go, but see profoundly increased tolerance from generation to generation.

I am for freedom of choice; it allows people and corporations to take action on which they can be harshly judged. Amazon has given publishers the choice of disallowing their author's works from being automatically read aloud by computer. Now, I actually have no issues with this. If an audio version of a work is provided at the same price as the print version, then we have some element of accessibility. However, if a publisher disallows this and makes no spoken derivative available I think I get the message: "fuck the blind." However, I'm no longer laughing.

New quintessential library for web engineers

nospam@example.com (Theo Schlossnagle) — Tue, 03 Feb 2009 23:08:04 -0500

At some point, I will come up with a list of "must have" books.

Until then, this reading list will do. I'm very pleased to see my work there amongst such exemplary company. There will always be a place in my heart for APUE.

Data explosion? WTF?

nospam@example.com (Theo Schlossnagle) — Thu, 29 Jan 2009 23:21:51 -0500

When choosing a database technology, "data explosion" is not really the image I want in my mind. Yikes!

Feb 2009 BWPUG Meeting: Stephen Frost, Column Level Permissions

nospam@example.com (Theo Schlossnagle) — Thu, 29 Jan 2009 12:42:15 -0500

Robert Treat was kind enough to coordinate the next BWPUG meeting. Here are his words, be there or be square!

UPDATED: the date of the meeting is 2009-02-09.

Welcome back everyone!

The last few months have been a busy time for everyone, but it's time to get focused again and get meeting again. To that end, for the BWPUG February meeting, we have lined up Stephen Frost to give a talk on Column Level Permissions in Postgres. Column level permissions are a new feature that Stephen authored which was recently committed into Postgres and will be available in the upcoming Postgres 8.4 release.

Stephen Frost is a Principal with Noblis, Inc., a non-profit organization working in the public interest supporting local, state, and federal Government. His work includes system architecture, system design, programming, unix administration, database administration, and management of the Noblis Innovation and Collaboration Center Computing Resources. As a PostgreSQL contributor, Stephen implemented roles support in 8.1 to replace the existing user/group system, and SQL column-level privileges in 8.4. As a PostGIS contributor and committer, Stephen updated the TIGER Geocoder for TIGER/Line and will be introducing a new version which works with the TIGER/Shapefile data at PGCon 2009.

Oh, and as a reminder, meetings are held at OmniTI world headquarters in Columbia, Maryland (http://omniti.com/is/here for details). We'll look to start around 6:30PM, and we'll have pizza and drinks available. We look forward to seeing everyone again.