Esoteric Curio

Remiss regarding reading recommendations.

nospam@example.com (Theo Schlossnagle) — Thu, 02 Sep 2010 21:26:43 -0400

It's been ages since I wrote about a book I've read. I didn't stop reading, just sharing.

I recently read Jared Diamond's "Gun, Germs & Steel." I'm not going to spoil the book, but not for the reason you think. You can't spoil this book without regurgitating it in its entirety: it is pure content. Every page is interesting and I refreshingly exercised my mind on the challenges of our world from an old-world perspective. If you've the time and patience (and haven't read it already), I highly recommend this book. Plain and simple, it makes you think. If you are like me (i.e. an entrepreneur or an engineer), you likely will think in atypical patterns. Thinking differently means thinking smarter tomorrow.

Enjoy!

If you've read it, please let me know what you (as a fellow engineer) thought of this book. Did you find it fascinating or is my head just malfunctioning?

Designing for vendor malfunctions

nospam@example.com (Theo Schlossnagle) — Wed, 18 Aug 2010 22:14:22 -0400

Many people have asked me how Oracle's recent actions will affect OmniTI and our clients. As you may or may not know, a considerable amount of OmniTI's internal infrastructure is built around the OpenSolaris platform. Given Oracle's recent announcement about their path forward toward Solaris 11, what does that mean for OmniTI and OmniTI's customers?

In short: what's old is new and what's new is old and business as usual.

The long of it has very little to do with Oracle and a lot to do with how we operate here at OmniTI. For a little over ten years, I've been on the international conference circuit speaking on the subject of scalable architecture design and implementation. While these talks discuss design patterns for scalable systems, they also strongly emphasize both strategic and tactical technology selection. One of the things I hope that attendees left with is a mission to maintain a strategy for avoiding technical lock in. A quick recapitulation:

If a vendor tells you (or you believe) that their solution is the only way to solve a particular problem, find another way. If there are multiple solutions to your problem and theirs is the "best," that is where the conversations start, business is won and relationships are made. Additionally, understand that the solution a vendor sells you is not for your problem; rather, it is for the portion of your problem that is shared by other companies in the target audience for that particular product. Armed with this information, apply some critical thinking when selecting technology. At OmniTI, we think critically… always.

Oracle recently made a decision to close the development processes around the future of Solaris. I am personally disappointed by their decision and it will necessarily mean that my participation in the OpenSolaris community will stop (as the community will no longer exist). While I am personally disappointed, I believe that their proposed business plan for Solaris 11 is sensible and will add more shareholder value over the short and medium term. As for the long term, the classic tech "long term" strategy applies: adapt to the market as it changes. Ellison has, in my opinion, been true and good to his shareholders. Back to how this affects OmniTI and our customers, we purchased Solaris 10 with support and we planned to purchase Solaris 11 with support. While the lack of nightly code drops and open development makes Solaris 11 less valuable, it is still valuable.

As with every technology selection we make, we reevaluate regularly to confirm that our decision remains correct. Additionally, we always implement systems in such a way that technology components can be gutted and replaced with relative ease. Could we transition our infrastructure from OpenSolaris to FreeBSD and/or Linux? Yes. Would the transition be relatively quick and painless? Yes. Why? Our job is to design, implement and operate systems that survive software malfunctions, hardware malfunctions, and vendor malfunctions. We've done our job by strategically designing our systems and maintaining the discipline in our tactical maneuvers that ensure our customers vitality even under changing circumstances.

Will we be moving our Solaris-based platforms away from Oracle? Based on what I know today, no. Will we be moving our OpenSolaris-based platforms away from Oracle? Maybe. We'll wait and see and cross that bridge if needed. Why such a casual attitude? At OmniTI we spend time throughout every project engineering and maintaining those bridges. When the time comes to cross one, it is a simple and casual stroll.

Reconnoiter at OSCON 2010

nospam@example.com (Theo Schlossnagle) — Sun, 01 Aug 2010 17:53:58 -0400

Reconnoiter: Large scale trending and fault detection

View more presentations from postwait.

Several people have asked me to post my OSCON Reconnoiter slides. Honestly, I was reticent at first because the presentation was packed full with oral content not in the slides and it had nice screencasts that end up turning into a crazy-large online movie. Suffice it to say, you need to attend to get real info out of it.

Despite my reservations, here is the PDF version of my slides with screencasts omitted.

If you want to see something largely the same (but with me talking at you and nice video) take a look at the nice production Miguel put together of my OSCON 2009 reconnoiter presentation.

Operating at Scale

nospam@example.com (Theo Schlossnagle) — Tue, 22 Jun 2010 14:37:44 -0400

As requested, here is my slide stack bereft of wit and cynicism:

Velocity 2010: Scalable Internet Architectures

View more presentations from postwait.

I just finished my presentation at this year's Velocity Conference. Thank you all for the warm reception and the positive feedback. This year's conference is going to be awesome. If you didn't make it this year make sure you sign up early next year; Velocity is not an event to be missed. If you want some more scalability goodness this year in conference form, consider attending Surge; it's going to kick ass.

Insight into the wild and crazy OmniTI

nospam@example.com (Theo Schlossnagle) — Tue, 20 Jul 2010 01:23:07 -0400

I recently had the privilege of being interviewed by Josh Ownes over at Webpulp.tv about all sorts of things. It's rare that I get time to sit down and think about all the crazy stuff we do at OmniTI let alone talk about it. Aside from a camera angle that is quite odd and very poor picture quality, the interview is really nice. While I typically don't watch or listen to my own media events (it's eerie), in this interview I actually enjoyed listening from start to end because it is a fairly fun list of wild and crazy things we to at OmniTI in unconventional depth.

Almost everything Josh and I talk about happened this year! And it's only half over! 2010 is really the "year of awesome" for innovation at OmniTI. We did cheat a bit by having several exciting things hidden in the oven nearing the end of 2009.

If you want to know what I do and what OmniTI does on a daily basis, I think this interview gives a lot of insight.

ZFS and Zetaback win again

nospam@example.com (Theo Schlossnagle) — Fri, 18 Jun 2010 12:33:55 -0400

Now that it's all set up, I gotta say, I think zetaback is the best thing since sliced bread for backing up big file servers.

We have an OpenSolaris file server with about 3TB of data, mostly in home directories. The kind of work my users do means that a lot of this data is in millions of small files. A full backup via rsync took a week; even a mostly empty incremental would take several hours due to rsync having to walk the tree and stat all those files. zetaback did a full backup in about two and a half days (mostly limited by the CPU speed of my backup server, since I'm using gzip compression) and an incremental took less than half an hour.

--
David Brodbeck
System Administrator, Linguistics
University of Washington

While I think that most of the accolades here go to the awesomeness that is ZFS, it is very nice to see that Zetaback has so elegantly made this magic accessible.

Congratulations to everyone what has made Zetaback what it is.

Surge: scalability matters

nospam@example.com (Theo Schlossnagle) — Fri, 04 Jun 2010 15:22:37 -0400

I've been attending conference in the "Internet space" regularly for about 12 years. I have enjoyed conferences about web technologies, open source, and programming languages technologies. Many, though not all, of the conferences I've enjoyed have been put on by O'Reilly — they put on some good shows.

Over the last two years I have immensely enjoyed my involvement with the Velocity conference. It is, in my opinion, the de facto conference on web performance and operations. Operations is about keeping it all running and web performance has embraced a definition of speedy content delivery and efficient browser execution. These two aspects of running large customer-facing systems are critical, but they are the bread of a sandwich and the meat is the "architecture."

So, where's the conference about how to architect systems that scale? I thought it might have been Structure, but this year's "buzz-compliant" cloud theme leaves me with a confident and resounding "no." Clouds aren't how you design and build scalable systems, they are simply one of many valid places "where" you build your design.

Fed up with all that, I went back home to OmniTI and asked "what can we do about this?" Our answer? Surge.

OmniTI has a reputation for scalable web applications and architectures. We didn't learn this stuff overnight. Like many of the success stories at Surge, we acquired experience through trial and error, constant collaboration between development and operations teams, and an unwavering commitment to excellence. But we still lean on our friends and peers to see how things can be done better.

Surge started as the brainchild of our employees wanting to bring the best and brightest in Web Operations to our own backyard...

I have to say that I have never before been so excited by a conference. Surge is the "how to scale" conference.

Vibram Four Fingers: Syndactyly Successful

nospam@example.com (Theo Schlossnagle) — Sun, 23 May 2010 21:21:39 -0400

These are vibram five fingers:

However, I have syndactyly affecting both of my feet.

I love the idea of having my toes separated which is why I prefer barefoot so much more than shoes. Vibram five fingers have always appealed to me. However, I cannot wear them. I wrote Vibram and asked for a pair of shoes to cut up and make acceptable for all the people with toes like mine, but they seemed completely uninterested.

This weekend, I finally took the risk and bought shoes I cannot even try on. And here is my short, simple and successful journey.

First I cut down the side walls between the second and third toes. I trimmed the fabric a few millimeters from the rubber sole and about four millimeters from the top seam (where grey meets black). Stopping here gives me wearable shoes that would collect rocks and offer a separation of the sole between the second and third toes that would be quite uncomfortable to walk in after a while.

The next part lack photography as the working space was very small and I couldn't take any solid pictures. However, I used grey embroidery thread to loop-stich the two toe soles together from the inside leaving the bottom attached:

Then I did my best at a plait stitch (which I'll admit I did poorly) across the top of the toes from seam to seam:

And then I can slip my darling feet into them:

All in all, this took about an hour for the first shoe and about a half hour of the second shoe. I'm pleased. It turned out well. These are my Vibram "syndactyly-enabled" Four Fingers.

Plan Wagon

nospam@example.com (Theo Schlossnagle) — Mon, 10 May 2010 10:54:33 -0400

What does it mean to architect a system? It means you solve problems.

While that might seem simple, I am absolutely dumbfounded by the number of people that attempt to solve their problem by simply applying the solution to someone else's problem without any sort of reasonable thought process.

Let me tell you all about "Plan Wagon."

Bob's a great parent. In fact most people that know Bob are extremely impressed by Bob's parenting skills. He spends time with his kids and one of his favorite things to do with them is take them for wagon rides around the neighborhood. Bob's blogged a few times about how challenging it was to take his kids around the block before (as they were all different ages with different walking speeds and attention spans), but now with the wagon — life is swell.

Jack reads Bob's blog amongst many others, and he himself has some experience with wagons and knows how they operate. He was a child once and just the other day saw someone transporting some children in a wagon. Jack is an important player in education: he coordinates various programs and has been very successful.

Jack is posed with a challenge at work: a new educational program being offered will serve 120 children ages 4 through 9 across a suburb and his organization must provide transportation. Jack feels confident. Jack knows Bob's success with wagons. Jack has used a wagon, has seen them work. Jack purchases a fleet of 64 wagons. He figures that two kids per wagon would work fine and that having four extra will leave some to be used if another wagon needs servicing. Now, who will pull them? Well, with 60 wagons, you'll need at least 60 adults. And the distance to be covered is about 4 miles, so they should allot for approximately 2 hours each way of travel time.

A casual observer notes to Jack that perhaps a transport with higher service capacity and faster travel time might be more appropriate. Jack is irritated because he knows that the wagons will work (and BTW, he is correct). Despite his irritation at being distracted from ironing out the implementation details of "Plan Wagon" he does some research. Jack finds that a both a Boeing 737 and an Airbus A320 will hold 120 children and go around 500 miles per hour. But, clearly those are impractical because of the cost.

A colleague of Jack's suggests that perhaps buses should be used. Jack, of course, knows exactly what these are. However, Jack doesn't have a C class license so cannot drive a bus, nor has Jack ever fixed a bus, and buses typically don't have seatbelts, which has always concerned Jack. Jack is confident that were something to go wrong with "Plan Wagon" he could act in any role required to facilitate success.

Jack sleeps on it. After reviewing the facts he comes to a decision. Jack's goal is to move different aged children around the suburbs. Jack knows that Bob effectively moves children around his suburb on wagons and the choice of a wagon made Bob happy, popular and solved the critically important problem of varying mobility of children of different ages that Jack is sure to face. Jack goes ahead with "Plan Wagon."

Jack is an ignorant ass clown.

Now, in reality, Jack's "Plan Wagon" can work. It's overly expensive and painfully suboptimal, but possible nonetheless. The issue here was very obvious: Bob's problem and Jack's problem were different — no matter how much Jack wished otherwise. For some reason, in technology, even seemingly smart people act like Jack.

People applying NoSQL to problems that would clearly benefit from deep relational management. People applying traditional DBMS to problems that require no atomicity, isolation or consistency. People who feel they need to use Hadoop to process a mere terabyte of data. These technologies aren't "wrong" — but can't be considered "solutions" when they are applied to the wrong problem.

Here's an idea: how about understanding the problem before you try to solve it? I know it's a radical concept, but it might just help. I would argue that almost every solution has at least two legitimate technology selections that can be cleanly applied. In other words, if I need a strong DBMS, chances are good that both PostgreSQL or Oracle will work. If I need a horizontally scalable key-value store, likely Cassandra or Voldemort (or MongoDB or CouchDB) will work. Now stop. I'm not telling you to look hard at the differences between Cassandra and Voldemort, I'm tell you to look hard at the differences between a DMBS and a key-value store. Do not assume that because you are building an app "today" that you it must be powered by a key-value store.

Next time you are posed a problem and you pick up your favorite tools — put them down. Why? It might just force to think about what you are building instead of the tools you are using. Be an engineer and think about the solution to the problem rather than the building materials and tools.

At OmniTI, we hire engineers and often immediately have them use a language and a database and an operating system they don't know. It separates the engineers from the programmers — programmers are a dime a dozen.

Unfortunately, engineers are hard to find. If you want to be an engineer consider OmniTI as a home for engineers. It was built by engineers and continues to be run by engineers. We're hiring people that are more excited about the solutions they produce than they are about the tools they use. We're hiring people that want to solve problems right and can restrain themselves from trying to jam a square peg in a round hole just because the peg is shiny or familiar. We're looking for people that care to understand a problem before they solve it. Ignorant ass clowns need not apply.

Regardless of who you are and what you are doing: avoid "Plan Wagon."

pg_controldata from SQL

nospam@example.com (Theo Schlossnagle) — Thu, 29 Apr 2010 09:45:48 -0400

Like many database, PostgreSQL stores critical (minimal) state about the database in what is called a "control file." This control file has valuable information in it that speaks to backups, checkpoints, block sizes, etc. PostgreSQL ships a tool called pg_controldata to dump this file's values in human-readable form. I've been frustrated in the past that you can't see all these values from within a PostgreSQL SQL session. At some point in the past I got in an argument about the usefulness of such a feature and I pretty well lost that argument: a postgres control file on an active database doesn't really show you (much) useful information and you really need it when the database is off (which is what pg_controldata provides).

PostgreSQL 9.0 changes the game. You can run queries on a database that isn't active (particularly a standby database that is applying WAL files). Now this feature becomes much more interesting. I can use monitoring tools with SQL-only access to find out extremely useful things about the state of the standby.

I have to say, of all the postgres extensions I've written, controldata had to be the simplest. Hopefully it is useful to someone other than just us.

from pcap to postgres: forensic performance analysis

nospam@example.com (Theo Schlossnagle) — Fri, 02 Apr 2010 09:52:07 -0400

Last week Robert Treat told me it sure would be nice if we could reconstruct PostgreSQL logs from network captures (in the sort of antagonist way that is: "MySQL can do it, why can't we?"). With pgsniff, we can.

Well, it turns out that he was complaining for a reason: a client. Our friends over at Etsy have a server that is so blindingly busy selling handmade things that logging all queries on the box degrades performance unacceptably. This is actually a common problem for PostgreSQL and other databases alike. Most databases offer a feature to log all queries where a single execution exceeds some arbitrarily chosen wall clock time. It turns out this "solution" is acutely painful because it often causes engineers to look at incomplete data and draw (you guessed it) incorrect conclusions.

The concept of a "slow query log" is a woefully incomplete concept. Typically, such a log is configured to track queries that spend more than X amount of time executing, where X is some "small enough number." Some sites put this at 1 second. If I execute a query (with any sort of high frequency) that takes one second to complete, I'm flat-out screwed. So, by the laws of physics, the queries that get logged under such circumstances are outliers (in high transaction, OLTP systems). The common approach to address this is to set X down to 100ms or 10ms. Any good engineer should realize that by doing so you simply change the resolution of the problem and have fixed nothing. The database behaves how it does because of all that it does. The queries you really care about are those that consume the most cumulative execution time, but to determine this you must account for everything. This is why turning on full query logging is essential to thorough performance analysis.

Now, most of my readers know I'm a DTrace fan and, yes, I've been able to do this in real-time, in production on all our Solaris-based PostgreSQL servers for years. However, pgsniff is a callout to all you guys without DTrace. This tool takes pcap information (from a live interface or post-mortem tcpdump).

The tool is a bit rough around the edges and some of the tuple tracking is buggy, but it is already in a form that is spectacularly useful. You can now tcpdump on a machine that can see the traffic (db, web server, monitor port) and reconstruct every query (even in more detail that PostgreSQL's logs provide). On the TODO list: fix bugs, production trials, and bind variable reconstruction so that one can dive into a slow outlier and see the bind variables used during execution of prepared statements.

For now, you can get the code via Subversion here: https://labs.omniti.com/pgtreats/trunk/Sniffer-Postgres/. Once some of the features are fleshed out and it has more production run-time I will put it up on CPAN.

pg_amqp slides from PgEast2010

nospam@example.com (Theo Schlossnagle) — Tue, 30 Mar 2010 00:55:40 -0400

This past week I had the privilege of presenting along side many distinguished speakers at this year's PostgreSQL Conference East 2010 in Philadelphia, PA. I presented PostgreSQL: meet your queue which was received even more warmly than I had anticipated. I really think that cueing your database to publish over AMQP is the bees knees and it turns out I wasn't alone!

Thoughts on the cloud

nospam@example.com (Theo Schlossnagle) — Tue, 23 Mar 2010 10:10:55 -0400

A couple of people have asked me to share my opinion on the cloud. I've been hesitant to talk about the cloud, because of the fanatics and funding; the hype makes people deaf to reason. So, instead of talking about the cloud, here's a technical discussion of cloud technology that attempts to dismantle the hype.

... what’s the problem? The hype. The hype is the problem. With hype come straw man arguments that delay or hold back the healthy evolution and incorporation of this technological paradigm.

Managing risk by deploying incessantly

nospam@example.com (Theo Schlossnagle) — Thu, 18 Mar 2010 14:07:13 -0400

I recently took the time to write down my thoughts on why successfully managing code deployments in an online architecture is so radically different from release management in a traditional software engineering endeavor.

...

Perhaps most challenging is the pace at which competition moves. In the online world, I can have an idea this morning, an implementation this afternoon and every client of my service that shows up tomorrow will see it. In fact, things can and do happen much faster than that. You might think that rapid concept-to-availability push is reckless. You might be right. But, your competition is doing it.

...

Squeezing precise numbers into fixed width types... or faking it

nospam@example.com (Theo Schlossnagle) — Fri, 05 Mar 2010 09:13:56 -0500

I have this app where we store numbers. Don't we all? Unfortunately, this app stores numbers without context. So, I don't know whether the number will be an integer or represent numbers throughout the real number space. Compounding this situation, I stand to gain particular advantage if I can store all these numbers in a fixed width datatype (each number consume the same number of bits of storage space). On current computer systems, there are two native types to each do some of what we want: the 64bit IEEE floating point type "double" and the 64bit integer types "signed/unisnged long long". However, I don't know which datatype to use before I actually see the number for the first time.

Let's take temperature as an example. In some accepted unit like C, F or K a possible value would be something like 37.23 or 49.05 which is a non-integer value where it would seem to make sense to use a float or a double. Round trip time in seconds is a great example of what one would think of as a perfect use for a IEEE floating point number as a float or a double. In that format, the most significant digits are more accurate, so if you are measuring the latency between two networked system (microseconds) or you are measuring the time to run an enormous data mining operation (days) you will get something accurate. Why is a type that looses accuracy okay here? If your operation takes months or years to complete an operation, you don't likely care if the operation took 2 years and 37 microseconds or 2 years and 12 microseconds. It may seem that a double would, generally speaking, be pretty good for just about anything. Not so.

In the real world, we frequently measure the number of octets (bytes) passed through switching equipment. These numbers are typically represented in a counter that ranges in integral value between 0 and 2⁶⁴-1. Now, if we store the number as a double, as we approach the upper end, the double starts to loose accuracy. Here's a quick sample so you can see it. I'll take X and X+100 as doubles and then subtract them to see the difference. As X gets very very large, 100 is less significant (the one-hundreds place), so we start to loose accuracy. I save you some reading by starting at 2⁴⁸ as that's where the interesting things start.

#include 
#include 
#include 

int main() {
  int i;
  for(i=48;i<63;i++) {
    uint64_t i1 = 1;
    double d1;
    i1 <<= i;
    d1 = (double)(i1 + 100);
    printf("%g - %llu = 100, saw %llu (%g%%)\n",
           d1, i1, (uint64_t)d1 - i1,
           fabs((double)((uint64_t)d1 - i1)-100.0));
  }
}

Producing the output:

2.81475e+14 - 281474976710656 = 100, saw 100 (0%)
5.6295e+14 - 562949953421312 = 100, saw 100 (0%)
1.1259e+15 - 1125899906842624 = 100, saw 100 (0%)
2.2518e+15 - 2251799813685248 = 100, saw 100 (0%)
4.5036e+15 - 4503599627370496 = 100, saw 100 (0%)
9.0072e+15 - 9007199254740992 = 100, saw 100 (0%)
1.80144e+16 - 18014398509481984 = 100, saw 100 (0%)
3.60288e+16 - 36028797018963968 = 100, saw 96 (4%)
7.20576e+16 - 72057594037927936 = 100, saw 96 (4%)
1.44115e+17 - 144115188075855872 = 100, saw 96 (4%)
2.8823e+17 - 288230376151711744 = 100, saw 128 (28%)
5.76461e+17 - 576460752303423488 = 100, saw 128 (28%)
1.15292e+18 - 1152921504606846976 = 100, saw 0 (100%)
2.30584e+18 - 2305843009213693952 = 100, saw 0 (100%)
4.61169e+18 - 4611686018427387904 = 100, saw 0 (100%)

You might ask, why the one-hundreds place is important in numbers that are this large. The answer is very simple. This measures octets through a switch, think of milliliters of water from a faucet. After you've used the faucet a long time, that number is staggering. However, if I want to know liters/second over the last 5 seconds it is very likely that the two samples I take (now and five seconds from now) will be both enormous and relatively close. And while the accuracy of difference matters, it is too insignificant to be represented in the IEEE floating point format. The interesting thing is, the source never reports anything but integral values, so if we were simply using an integral datatype we would have exactness.

So, what do you do when you don't know? Arbitrary precision math, or in SQL: numeric. Numerics are great because you always get the correct answer. Numerics are bad because computers are not very good at doing math outside of the IEEE floating point and integral spaces; in other words, it's painfully slow. Perhaps worse than being slow, the numeric datatype is a variable width type, which in this case means I've lost the battle.

Now back to my app: the inputs are either floating point numbers or integral numbers, but the only way we can tell is if they contain a decimal point when expressed in non-scientific notation; or, do they have any non-zero values in places to the right of the one's place? What we want to do is give accurate integral storage to apps that are courteous enough to provide integral numbers in the range of -2⁶³ to 2⁶⁴-1 and use a double otherwise. This allows us to use 64bits of fixed space to store the value (though we need some extra bits to note that the 64bits represents a double vs. a signed integer vs. an unsigned integer). Once we've tackled this, we can accept numeric values from SQL stored them in fixed size (with expected loss of insignificant places for floating point and expected accuracy for integral values) and return them back as numerics.

My little project fronts against PostgreSQL, so I had to convert integers and doubles into numerics. Much to my surprise, the internal method of converting this is to convert from type1 to a string and then from a string to type2. This is a very inefficient process. It turns out that direct double to numeric conversion (as expected) is much faster. In our case it shaves off approximately 50% of the CPU cycles.

As a disclaimer, this is a very special purposed storage type and neither replaces the double, integer or numeric type. It is, instead, a hybrid type with special properties that are attuned to the problem at hand. Before you go implementing something similar, make sure you know what you are doing. Our goals were to retain accuracy for integral types, achieve a fixed-width storage format and still allow representation of numbers both large and arbitrarily smaller than 1.

What does it mean to be a patriot?

nospam@example.com (Theo Schlossnagle) — Fri, 26 Feb 2010 21:07:15 -0500

I love my country. My sense of patriotism has grown since I was a child to what it is today. I can't imagine being a citizen in any other country (I hope all foreign nationals feel the same about their country). What does it really mean to be a patriot?

Patriotism is support of your country... I believe my country was founded on a principal of liberty. Liberty is the state of being free from oppressive restrictions imposed by authority on one's way of life or behavior (or political views). "Give me liberty of give me death" is a very famous American quote that deeply represents what the United States of America is all about; Patrick Henry was no chump.

Liberty means that you never sacrifice your rights or freedoms for simple conveniences such as safety. In plain English, the US PATRIOT act is the most unpatriotic American act in my lifetime. Damn sad. I am a patriot, I will outlive this abomination and I will once again live in a country where our citizens and our representatives act out of principals instead of a sad fear and artificial sense of safety. It only requires staying the course and acting without fear.

Death by icicle.

nospam@example.com (Theo Schlossnagle) — Fri, 12 Feb 2010 21:59:34 -0500

This fell off the gutter onto the doorstep. Damn lucky no one was standing there. Just ridiculous.

Sexified fonts for the web.

nospam@example.com (Theo Schlossnagle) — Mon, 11 Jan 2010 15:08:05 -0500

OmniTI has been working with Clearleft for a while now on Fontdeck. The super Greg Chiasson has been pushing our beta live and I've been provisioning machines and setting up infrastructure just so you can read this web page in a nice sexy font... Nice... Sexy.

And yes, of course, fontdeck is powered by our friends: Apache, PostgreSQL, OpenSolaris, a bit of Linux here and there, a sprinkling of both asymmetric and symmetric cryptography and an ancient dialect of computer-speak called Perl.

Asynchronous PostgreSQL Candy

nospam@example.com (Theo Schlossnagle) — Sun, 27 Dec 2009 13:49:32 -0500

I've put the first (intended) use of the pg_amqp setup to the test. So far I'm very pleased. While none of the code for the usage is open source the real "magic sauce" is open and has had a few bug fixes and bits of robustness added since I last posted. I can, however, describe the use and let your imagination run while.

The parts: PostgreSQL, RabbitMQ, and a bit of Java around Lucene. Our java program actually uses jetty to expose a tiny servlet that accepts search queries and returns json results making AJAX-style searches easy and fast.

The obvious question we all face in these sort of configurations is "how do keep our search indexes up-to-date?" Usually, we relax constraints (wisely so) and determine exactly "how up-to-date does out index needs to be." If you can answer this question with a number sufficiently far from zero, you've won. In our situation, the user experience must be able to search and find the updated items in database via search immediately after insert or update (within a 100ms or so).

Enter message queueing. The first thing we do is make our Java process connect to RabbitMQ, bind a queue and consume. The messages it consumes from the queue have instructions on the precise element that has changed which then causes an query against the database retrieving all the new data to be reindexed (in our case, it is much more than can be easily witnessed from a trigger on update) and updates the Lucene indexes.

The second updated we made is to PostgreSQL by installing pg_amqp. On the tables whose changes should induce reindexing, we add a (or augment the existing) trigger to call: amqp.publish('amq.direct', 'searchstuff', E'reindex\t' || NEW.rowid). Assuming we have a column called rowid, this will queue a message that looks like "reindex<tab>328432"

The neat part is that if the update is rolled back in the database, the message is never sent. Otherwise, we see the message from the our indexing and search app and viola.

I think this technique is really useful for people that are currently leveraging Lucene-based systems to provide powerful search functionality yet keep their data safe and secure in PostgreSQL.

OmniTI seeks data management experts; a new type of DBA.

nospam@example.com (Theo Schlossnagle) — Thu, 17 Dec 2009 11:49:25 -0500

What's a DBA? Database administrator? Database architect? Whatever it is, we need one (or two or three) at OmniTI. The problem is that the Jurassic variant of the DBA simply can't cut it in today's world of the web giants (OmniTI services the web giants). What is it about the modern use of database in the web world that make traditionals DBAs so inadequate? It's actually a variety of things. If you think tackling them is a challenge you're cut out for, send me your resume!

Scale

One might say "Scale? Scale? Are you kidding me? Traditional database problems come in various flavors of gigantic." This is true, but the unique dynamics of the web mean that not only is the data flowing in at uncontrollable rates, but tomorrow there is always a new data source with a new representation. Due to the scale of external systems and your own it makes for an interesting sandbox in which we play. A multi-terabyte database is quite common just about anywhere these days; however, ten of them running four different data management platforms is rather common place within large web infrastructures.

Continuity

This is the real game changer. This is the mental paradigm shift that classical DBAs struggle with most. Try this on for size: "no maintenance windows." Okay, admittedly that is a stretch, but only a small stretch. How about 5 minutes of database downtime allowed every 4 weeks or, worse, every 52 weeks.

Coupled with this is daily (or more frequent) code rollout often times requiring database schema change, new triggers, and different replication dynamics.

The amount of seemingly ass-backwards engineering required to accomplish this feat is usually described as either: "excruciating pit of hell" or "fascinating enigma." (I think of it as both) What I do know is that I have seen some of the most profound data engineering creativity emerge from this insipid constraint.

Innovation

While noSQL is a fanatical movement that needs to be kept in check, the problems it addresses are real and [some of] the solutions promoted are good. Understanding when data better fits in a key-value store is something that the classically trained DBA struggles with. It's all about data management: Dynomite, CouchDB, Cassandra, Oracle, PostgreSQL, MySQL… they all have a sweet spot &emdash; know it.

Scope

The other interesting dynamic has less to do with data management and has more to due with the pace and culture of web companies. In traditional companies, roles are more disjointed; DBAs have a set of responsibilities that software engineers and system administrators do not have and those two groups have a set of responsibilities with which DBAs need not concern themselves. This "silo effect" spells absolute disaster for web companies.

The pace on the web is simply too fast (I often think it is a bit reckless, but that's another rant) to keep all the parts separate. An insufficient amount of time exists to define how one layer of an architecture will communicate with another layer. And while the layers meet at reasonable places most of the time, there are other places where the lines are blurred. These are the points in the architecture that stand to have the highest efficiency, but also are the hardest to troubleshoot; they require an organization of engineers that have no boundaries. In the case of a DBA… one that wants to look at the ORM code that builds the insanely awful SQL queries that beat the stuffing out of their database every second; one that is actually aware of the storage configuration and layer-2 fabric that sits behind the magical "remembering device" that the database commits things to all day long.

Beat you with a stick?

Around here, Robert Treat runs the show. The stick wielding and beatings are left to him (he a father of three, so I'm sure he's more than capable).

I guess the question is… are you up for the challenge? Are you a DBA who thinks their job is about more than just ACID? Are you a developer or an SA that has decided to make a shift and tackle the data management layer with a unique perspective? Are you interested in mind bending problems that arrive on your plate a bit too quick for anyone's comfort level? I hope so, because we need people like that!

AMQP for PostgreSQL

nospam@example.com (Theo Schlossnagle) — Sat, 19 Dec 2009 18:00:27 -0500

So, it turns out that being stranded in an airport can lead to some productive output after all.

I've hacked together an AMQP extension for PostgreSQL. If you don't know what AMQP or PostgreSQL are, stop reading.

One thing I've needed to do for a while is be able to submit a message to a message queue from within a PostgreSQL transaction. However, obviously (because we run a real database here), if the transaction aborts I'd rather not have those messages sent. Enter pg_amqp.

pg_amqp allows:

Multiple connections to different AMQP brokers (configured in the amqp.broker table).
Allows for the declaration of an exchange.
Tries to be smart about bring up a connection to a broker on demand and leaving it connected to accelerate the next publication request.
Allows AMQP publishing from within SQL that is transactionally aware (only publishes on commit).
Allows AMQP publishing from within SQL that is autonomous (happens outside the current transaction).

This is completely alpha. I've tested it quite a bit by hand, but put no load on it whatsoever. Monday of next week, I'm going to beat the ever-living @#$% out of it on one of our systems that desperately needs on-commit AMQP notifications.

I'd love some more eyes on this. It has some flaws, specifically related to processing asynch events from within a PosgreSQL backend (which has no concept of asynchronous even notification). As such, if you do stupid stuff, stupid things happen. The easiest solution is to add a thread to each postgres backend to process the backqueue of events from AMQP; however that so blatantly violates postgres' process model and design that I've no intention to do that. With all the issues aside — it works pretty well for me.

I didn't add queue declaration or the ability to consume from queues. That was intentional because PostgreSQL can't run SQL outside of a transaction block making a while(1) { consume, do, commit } impossible from within SQL itself. Without the ability to do that it seems really useless (and pretty dangerous) to allow someone to do a blocking AMQP frame receive from within SQL. If you think it would be useful for you, let me know — it would be a trivial addition.

Oh yeah: pg_amqp

Jive turkey...

nospam@example.com (Theo Schlossnagle) — Thu, 19 Nov 2009 22:26:12 -0500

So... lock a dude on a flight with Internet access, free booze and free movies... yet, he codes. Nothing useful, of course.

One of my favorite programs from past days was the jive.l lexer (it's an Internet scavenger hunt to find it). Well, here's a jive bookmarklet. Try it out. Boredom on flights can be dangerous.

Itch scratched - right between the rabbit's ears.

nospam@example.com (Theo Schlossnagle) — Fri, 13 Nov 2009 17:18:32 -0500

Net::RabbitMQ, get it while it's hot.

Extending and Embedding

nospam@example.com (Theo Schlossnagle) — Tue, 06 Oct 2009 22:21:48 -0400

If you’re like me, you’ve read a handful of really useful articles about extending and embedding things like perl, Python, Java, lisp, scheme or lua. The slant there is a technical one: “So you need Java, here’s how to embed it.” I’ve embedded Python once, perl and Java countless times and, most recently, lua. Throughout this I realized that I don’t need the languages as much as I need a convenient extension out of C. In other words, if your foundation app is feature-rich, embedding isn't as much about adding the features of a specific language as it is about exposing the features of your system.

Given this context, there is a lack of articles about how to choose the appropriate embedding technology for a given project. Each language has its own advantages and people religiously argue about them seemingly to no end. Things get interesting when you look to embedding languages, as their internal implementation decisions start to influence what is and isn't possible. I’ll take you on a brief whirlwind tour of my personal experience embedding.

Perl

Many of you know I program in Perl quite often. Many people hate Perl and believe it to be an abomination. I love Perl for its flexibility, expressiveness and its terseness — aligning it with myself. However, the perl (5.10 and earlier) interpreter is a horrid, catastrophic example of how not to write a virtual machine for an interpreted language; spaghetti does not begin to describe that mess that is perl internals. The API is insufficiently documented, the internal data representations are indecently exposed and the the way thread support was backed into the code (like a garbage truck through a toy shop) make it more than a challenge to embed. Now, I’ve been writing XS code for years (the C-like language for extending perl itself) and I know and love the language itself, but I’d argue that it is rarely a good idea to embed perl in any application.

You may have noted that I claimed to have embedded perl several times in applications. Some of those turned out to be a good idea. Both (yes only two) of those times, the reason it ended up being a good idea was that to make my application more useful I needed the wealth of CPAN (the comprehensive Perl archive network that contains a wealth of easy-to-use Perl wrappers around common needs). While in the end I won, embedding perl was an awful experience and I wouldn't recommend it to anyone. My specific problems came down to debugging memory usage as it transcends from C into Perl heap and the painful problem that is threading within perl. Each time, integrating into perl’s threading system proved a waste of time and I ended up starting an interpreter for each thread (or had a resource pool of interpreters from which each thread would reserve). So many useful CPAN modules aren’t thread-safe and were never developed with a mindset that they would be loaded into a perl that might be embedded in another system.

While I love the language, I hate the implementation.

Python

While I like Python as a language, I personally find that if my job doesn’t fit in perl and that Python would be a better choice, I usually think a bit harder and realize that C would be even better. I realize this isn’t the same mental leap made by most people, but I love C. C is my favorite language. In fact, I wouldn’t be writing this article if I didn’t write so many applications in C that can benefit from an embedded interpreter to ease configuration and processing tasks.

In fact, I would argue that Python is a great language to embed because the of the interpreter. It's implementation is cleaner than perl’s — as I’ve mentioned the bar could not be set much lower. The fact is that embedding Python in a complicated multi-threaded and/or even driven application really shows the inadequacies of its embedding design. That combined with the fact that I’d rather be coding directly in C means: "I wouldn’t embed that again."

Now, all the applications in which I typically embed interpreters or virtual machines are complicated. If your application is single-threaded and doesn’t do extensive co-routines (e.g. closure-oriented programming or event driven systems) then Python could really be an ideal embedding language.

Java

Java is a complicated discussion. I have a lot of mixed feelings about Java. No language should have functions in its core distribution deprecated at the rate that Java boasts — it’s dysfunctional. Java as a language itself is quite nice. It is powerful, fast, and expressive. It has some annoyances like its verboseness and sprawling code bases. It has one true, deep negative: Java programmers. Now, not all Java programmers are bad, but I've seen far too much Java code produced by programmers far and wide that is of a quality that is flat-out unacceptably low.

When you’re embedding a language, some of those negatives don’t matter so much. We can instead concentrate on ease of embedding. Java has to be, hands-down, the easiest system to embed. The API is crystal clear and absolutely complete. It is all held in a single, small header file called jni.h. Every time I’ve embedded Java is has been simple and painless. JNI team, pat yourself on the back — I’ll happily buy a round of beers if we ever run into each other.

On top of the ease of embedding, the native support for mapping application (C) threads into JVM threads is concise and demonstrates pre-meditated good engineering.

One reason I don’t like Java for this is that it isn’t really “interpreted.” I have to compile (or auto-compile) my code to get it to run in my app. Now, perhaps I could use Java to pull in something like Jython or JRuby to make a rather obtuse, but convenient, system. Success via indirection has never been high on my list — if I ever try it, I’ll be sure to let you know how it went.

Tcl, lisp, ruby, PHP

Too old, too arcane, AYFKM, leave it on the web guys, respectively.

Okay, in all fairness, I didn’t give Tcl its due. It’s likely the most widely embedded language. I just don’t like Tcl and it shows.

Javascript

Javascript deserves its own section for arguments against embedding a language. If you have a single-threaded program that is not event driven, Javascript is a brilliant choice. I don’t remember the last time I wrote an application that was both that simple and was in need of an embedded language. So, the fact that Javascript has no support for threading makes it utterly painful and obtuse to embed in most systems. This saddens me. Of all the languages I program in, Javascript is likely my favorite. I know this must be on the merits of the syntax and expressiveness of the language because I loathe DOM and anything that touches it. Oh! how I wish Javascript embraced threading. Oh! how I wish Javascript had the embeddability of Lua.

Lua

Lua is my new love. It is the reason I decided to write this article. I actually don’t like the lua language. I don’t particularly like the internal implementation (garbage collected instead of ref counted). It doesn’t have particularly useful extensions to add feature value to your system. So, what is it about lua?

I᾿ve been working a lot on a system called Reconnoiter. noitd, the agent responsible for performing active checks against other systems, has a high-performance hybrid thread/event core. Writing code for a hybrid thread/event system can be quite mind bending to write and challenging to debug. Reconnoiter allows writing checks as modules that are dynamically loaded into the system at run-time. The hybrid thread/event core means that the core itself is capable of handing hundreds of thousands of open sockets (think network connections). Writing complex event-driven C code severely limits the audience that can contribute check code to this open source project. This frames the need.

Managing hundreds of thousands of concurrent threads in the system is a recipe for suboptimal performance. In order to remove the complexity of event-driven programming, I need to extend Reconnoiter’s core to a language that could provide the feel of a procedural (or OO) programming atmosphere while maintaining a non-blocking, continuation based implementation for high single-system concurrency.

Now, this is possible (sort of) with many of the embedding choices. When I get to a point in the interpreted code where I need to suspend what I’m doing and complete it later when actionable data is available (say date from a network read) to take some action. The issue is that in order to resume all these languages where they left off, I need to maintain their C stacks. This can be accomplished with a user-space threading implementation strategy using setjmp, longjmp and a healthy sprinkling of black magic. In a lot of ways, this approach doesn’t solve the problem if requiring a interpreter or VM thread for each concurrent operation which would voraciously consume resources.

Enter lua, champion of concurrency. lua is a stack based virtual machine specifically designed for integration with larger C systems. Within lua, assuming you don’t use any third-party extensions (of which there are few), I can leverage the lua_yield and lua_resume API calls to suspend the execution of a lua co-routine (lua-space thread) without the need of capturing a C stack. The state of lua’s execution is represented in the lua stack and simply pick up later right were I left off. This is a technical nuance, but one that provides a considerable amount of freedom. In a single thread, I can open 100k network sockets and have them all independently driven by lua programs without maintaining 100k C stacks. Furthermore, the lua programs give the feel of blocking reads and writes making network programming once again bearable.

After wrapping Reconnoiter’s libeventer and check systems with lua, I was able to replace the previous incarnations of our HTTP checker (based on libserf, then on libcurl) with a 170 line lua HttpClient that consumes less memory and less CPU! If you’re wondering, yes, the client supports chunked transfer encoding, gzip/deflate content encoding, arbitrary headers, client payloads and methods.

So, despite not really liking lua as a language (I find its syntax a bit painful), the simplicity of embedding it my application was on par with Java, the fact that it is truly interpreted (no compiling lua before it will run) and the absolutely brilliant exposure of its continuations as first-class embedding APIs puts me in bed with lua.

Summation

Never embed a language just to embed a language; always have a purpose in mind. Understand the host architecture in which you need to embed and attempt to match the strong points of a particular interpreter or VM to your needs. For example, if you have a heavily threaded host architecture, don’t embed something that doesn’t support threading well (or claims to but takes global locks).

Next time I embed, I will start with three preferences: Java, Javascript and Lua. Sadly, I predict Javascript will be disqualified quickly due to its deficiencies. Java and Lua both have that pros and cons. I just hope the next project finds Lua the best bedmate, I’m already looking forward to working with it again.

A look at rubyrep

nospam@example.com (Theo Schlossnagle) — Thu, 20 Aug 2009 15:27:12 -0400

rubyrep looks neat. I suppose our next round of tests will have to use it on some billion row tables and see how it fairs.

highscalability.com talks about OmniTI's Reconnoiter

nospam@example.com (Theo Schlossnagle) — Sun, 16 Aug 2009 15:13:07 -0400

It’s exciting to see things starting to take off. The momentum and excitement around Reconnoiter are at an all time high here at OmniTI. I really appreciate Todd Herr’s extensive write up about Reconnoiter over at highscalability.com — the review was spot on.

It has ample criticism (all deserved) and we’re continuing to work on the “consumability” aspects of the product. One of the comments he made was the interesting choice of Lua as a language to extend the product. I really wanted to use Perl or Python to extend Reconnoiter, but Lua has some pretty magical properties that made it meld well with the noit internals. In my opinion it is perhaps one of the more technically interesting parts of the product as the search was long and the choice reluctant. When I have more time, I’ll write an article about embedding lua in noit and the deeper reasoning behind it.

Also, don’t be scared by Lua, you can always write extensions in C! Although it doesn’t meld well with its high performance design, there is an extproc module that allows writing checks as standalone scripts and operates fine with existing Nagios checks — I say I wrote it to make adoption a bit easier, but in truth Eric and Mark made me do it.

Big Bad Postgres Indeed

nospam@example.com (Theo Schlossnagle) — Mon, 10 Aug 2009 21:45:39 -0400

I gave a talk at the Percona Performance conference (same time as MySQL, in the same facility... can we say awkward?) about running large PostgreSQL installs. I referred to a few instances in the presentation that are a handful of terabytes in size. In today's world, these aren't that large, however we do pretty deep analytics on these installs. It is most definitely not a case of store and forget.

A few people came up and said: "I thought you were going to talk about big... a terabyte is not big." I would rebut that with it's not how big it is, it is what you do with it, but then I would be on the defensive. The truth of the matter is that it is a combination of things. Size matters: below a certain threshold, you simple can't call it large. Usage matters: if you don't do something interesting with the data, you might as well be throwing it away. While most of the PostgreSQL instances we deal with were considered larger by 2005 standards, a terabyte simply no longer meets the bare minimum for "large."

I'm excited that we're looking to launch a new service that will turn the tables on large for PostgreSQL. We very well could have 10 petabytes in postgres in no time if things go as planned. Not only will we meet the size requirements, we'll also be doing lots of interesting things with the data. Big Bad PostgreSQL indeed.

Reconnoiter and another platform

nospam@example.com (Theo Schlossnagle) — Fri, 27 Jun 2008 17:27:27 -0400

Reconnoiter is coming along. Unlike most open source project, I tend not to talk about mine until their are really useful to people. Over the last year, I've adopted the unhealthy attitude that useful means "shiny front-end." So, I'm blogging to break that attitude and talk a bit about project that doesn't have a shiny front-end... yet.

Reconnoiter is built out of years of frustration using tools like RRDTOOL, Munin, Cacti, ZenOSS, Nagios, etc. etc. I have a lot of problems with these tools. First, they are not efficient. I need a powerful machine to monitor a mere 10k services. And it actually gets to be an engineering challenge to monitor 100k services with these tools. Also, the graphs are about 10 years old with respect to design and usability. I want something new, something fresh, and something that doesn't need a damn web UI to configure. Several people have asked, why are you reinventing the wheel? Why don't you just improve an existing product? My answer is that I want a well-thought-out product foundation so that I can trust all the bits. I want reponsibilities decoupled at the right spots. I want data in a form that the world can query and run reports the likes of which I have not concieved. I don't want the load on my monitoring machines to be 8. I want my monitoring system to check services and metrics when it planned to, not several minutes (or even 2 seconds) after it told me it would. Simply put, I expect it to work well, all the time. And, of course, I want it to work how I would expect it to work.

Reconnoiter was born out of the need to monitor the internals of many disconnected data centers with between 10 and 1000 machines in each facility. Monitoring can mean a lot of things, here I consider it to be the collection of metrics and awareness of their availability. In and of itself, monitoring is pretty useless, but it is the foundation for two critical pursuits in Internet infrastructure and business management: fault detection and trending.

Fault detection is as simple as understanding when something has faulted. However, knowing something is broken is easier than knowing something is about to break. Is it better to know that your machine just crashed because the chip slagged to the motherboard, or that the temperatures in rack 043 are rising unexpectedly? Answer: both, but I hope I only learn the latter and not the former. Truly, there are too many things to monitor... hundred or thousands of metrics on each piece of equipment. I can't reasonable go in and configure good/bad thresholds on each one. I want anomaly detection. I want a system that I can say: "this looks right, tell me when it stops looking right." That, to me, is a much need companion to tradition fault detection.

To me, trending is much more than drawing graphs... it is about intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you fucked up getting where you are now -- in the vain hope that you learn from your mistakes and plan better next time.

Reconnoiter is an attempt to build these things. Building a system requires starting with pain (need), solid structure and plumbing (good engineering). So, reconnoiter is underway. And this post is in mid-step:

It started on OpenBSD, and added support for FreeBSD, Mac OS X, Linux.

As of changeset [292], we have Solaris/OpenSolaris support.

We have a pretty nice front-end for trending under construction, but it isn't there yet. We'll have numeric data combined with textual "event" data on the same graphs. All that convenient stuff. Here's the rather plain-Jane graph you get now (because some people won't even read a post if it doesn't have a pretty graph):

Honestly, I don't know what the value of this post is, but people around here keep telling me that people should be aware of an open-source tool like this, even if it isn't finished (read: usable) yet. I say it isn't usable yet, but on our development instances here, we monitor 2892 production metrics across two data centers and the load never peaks past 0.10. I'm pretty excited about where this is going. Honestly, my favorite part right now is that I can configure and control the noitd checking nodes via a telnet console and it acts as if it is a piece of network equipment rather than an "application" -- as it should be IMHO.

Painful reading.

nospam@example.com (Theo Schlossnagle) — Sat, 18 Jul 2009 21:50:14 -0400

I just read “Selling to Big Companies” by Jill Konrath.

“Ugh.” Written on a third-grade reading level with a tone of voice often focused on a fourth-grade audience this book was beyond painful. The author adumbrates a plan, but (being a decision maker) I find several of her positions paradoxical. The book does has useful information, but it was not reinforced with compelling storied or anecdotes. As I trudged to the end of this book I realized that reading just the “Key Points” sections at the end of each chapter just might have ameliorated the experience.

Basic premise: good decision makers (like those at big companies) buy on business value, not on shininess of product. Don't sell your offering, sell the value they will realize. Understand the company and personalize the pitch. All of this should be obvious to anyone who is even considering taking a meeting with a large corporation. Unless you are selling commodity services, the shotgun effect never works well in sales. For specialized services, the shotgun effect can be a good marketing approach (if done well) — never a sales approach.

The section on words to leave out of email correspondence due to spam filters was simply uneducated.

It hurt. This blog post is only a method of me expelling my disappointment.

Only buy this book out of morbid curiosity of what I can endure reading.

Web: you can now not suck.

nospam@example.com (Theo Schlossnagle) — Thu, 25 Jun 2009 23:33:33 -0400

In perhaps a new trend, I’m blogging from 39011 feet (or so says the seatback in front of me). I’m traveling back home to the east coast from San Jose, CA where I attended (and spoke) at this year’s O’Reilly Velocity Conference.

I participated (and blogged) about the Velocity Summit in which I’ve participated for the past two years. The summit is the unconference preceding the real conference that help the organizers digest current hot topics and better define the conference track for the actual conference. The summit itself is filled with enough brain power to warp space-time, so I drop everything to go to that.

Ironically, despite being a well respected authority in web site (and general internet) scalability and performance, my talk proposals for Velocity 2008 were not accepted — I clearly need to write better proposals. This year, I managed to work my way into the workshop track on Monday. Despite having a bad headache and feeling "off" the day before, I managed to get my act together and put on an A-game for my workshop. For those of you interested, here is my scalable09 slide stack.

I thought I’d take a moment to talk about what I liked about the conference and what I think could use some improvement. I realize this is a down economy and that might be a legitimate justification for some the actions that resulted in some of my disappointment.

First, the negative. I usually start with positive and end with negative because I’m a pessimist. However, all in all the conference was awesome, so I thought I’d get my short list of gripes out of the way early.

O’Reilly is infamous for throwing good conferences for geeks. In my opinion, the field of web operations has been so severely neglected and applies so broadly to the world today that this conference needs to be for everyone.

In the next conference, I’d love to see a technical business track. Several of the talks I went to spoke to the dollars and cents lost or earned by paying the right amount of attention to web site performance and better operational paradigms. I thought a lot of the topics would be very useful to business managers.
The first day was not video taped and the second and third day were only half video taped. Come on guys, ante up. The attendance fee was substantial, you can afford to give your attendees the value of watching what they had to choose not to attend. I like the option when I go to a conference to choose a session that seems interesting so that I have the opportunity to participate, but often times I find that another session was top notch and I expect to be able to later review a recording of that.
Lastly, and this is the most significant. While I thought the conference was extremely well executed (excellent job Jesse, Steve, all your support, and most definitely O’Reilly), it lacked sufficient PR and marketing outreach. I talked with several journalists (as a part of my normal day job) while I was at that conference and not one of them was aware of Velocity — simply embarrassing. Given that the Structure conference was in town that same week, O’Reilly should have invested more in their PR and marketing outreach. It would have resulted in a substantially increased audience and a better venue for teaching the world to run a faster web.

Now that I’ve griped and aired my disappointment. I can focus on the gobs of awesomeness that was Velocity.

The conference was put on quite well from an operational perspective. Things started on time, A/V problems were non-existent. Like an idiot, I managed to lose my MacBook Air power adapter and the A/V crew managed to recover it for me. Conferences just plain suck when they have technical difficulties; this one had none.
The two tracks at the conference were extremely well articulated and while I wanted to be in both all the time (as OmniTI is a full-stack company, we care about both equally) it was an excellent split.
One track was performance which focused intently on user-perceived performance. This was largely front-end (HTML,CSS,JS,etc.) but also had a healthy amount of deep stack performance discussion as well including the often ignored, but much deserving networking aspect of delivering the web to users.
The second track was operations and I feared it would be bunch of blind “The Cloud Solves All Our Problems” sessions. Much to the contrary, it focused heavily on operational strategy and and what it takes to execute tactically. There was a bit of cloud here and there (okay everywhere), but there was very little blind and ignorant mentality that launching in the cloud was a solution to a hard scaling problem.
I had the opportunity to see a lot of old faces, but spent most of my time meeting new ones. The opportunity to learn more about other people’s problems is what completes me as an engineer. The people I met at this conference were both honest and open and provided a fabulous and refreshing perspective on what today’s performance and scalability problems really are.

In my workshop, I spent about 20% of the time discussing the philosophy of being a good engineer and 80% discussing practice (non-cookbook) with examples and advice. The basic message is that systems are complex and you must think of all the parts holistically or its a recipe for disaster — or failure.

Two of my favorite talks were Nicole Sullivan's “The Fast and the Fabulous: 9 ways engineering and design come together to make your site slow” and “ 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” by John Allspaw and Paul Hammond. While Nicole's presentation, like mine, was not recorded, the other was and if you want to break down the divide between operations and development, it is a must see.

All in all, I would encourage everyone reading this to attend next year's Velocity conference. I am certain you will walk away with knowledge that is both valuable and applicable.

Business Chaotics, It's All I Know.

nospam@example.com (Theo Schlossnagle) — Sun, 12 Jul 2009 10:02:57 -0400

Yup, I read another book. I think we can all agree we live in turbulent times. Current economic conditions combined with market globalization causes all sorts of fluctuation and chaos. I enjoyed parts of this book, but didn't find it particularly valuable. I believe the lack of value is not the author's fault, but rather a condition of me being a young entrepreneur.

I liked how the author reinforced the concepts by correlating turbulence and chaos to our immediate past (2008). What unfolded in 2008 is still quite fresh in my mind and it forced me to draw more meaningful parallels into the book. While we've had bad economic times in the U.S. before, the last time it was really bad I was only nine years old. I don't have scars from that. The harsh realities that hit in 2008 (and 2009 at least) were more directly meaningful to me as a business owner and employer.

"Chaotics: The Business of Managing and Marketing in the Age of Turbulence" by Philip Kotler was really a discussion to help people mentally transition from an old paradigm of business management to a new one. In the old one, there were good times and bad times and the transition between the two was, while not exactly predictable, regular and rather slow (measured in multiples of years). In the new paradigm, there is a realization that shit happens and it happens all the time. Due to globalization and other factors, there is a lot more chaos. In the old paradigm people were taken off guard by the unexpected events of natural disaster (weather, fire, etc.). Now, regular economic fluctuations can take people off guard. Globalization, hypercompetition and the fact that we (as a global community) apparently don't understand how to manage risk well have introduced all sorts of uncertainties.

Business chaotics is the mindset and process of preparing for uncertainties and having plans laid to deal with the shit that is sure to happen, even if we don't know precisely kind before we step in it.

So, why did I say I didn't get much value from this book? Don't get me wrong, I got some... However, I didn't have any revelations as the reviews indicated I might. The reason for this is that I've been an entrepreneur and business owner in the Internet sector for 12 years. My company enables start-ups and other companies intending to serve a global audience on the Internet. We apply technologies appropriately to improve the experience of their users (and thus increase revenues) as well as reduce operational costs. This marketplace has been turbulent since its inception. This is the only type of business environment I know. As such, this book didn't help me transition mentally from the old way to the new way because I don't know the old way.

Project Management Reading

nospam@example.com (Theo Schlossnagle) — Sat, 11 Jul 2009 23:29:26 -0400

Long ago I studied Project Management very briefly. OmniTI does a mix of project work and operations work and the orchestration of those two things is quite interesting (more to come on that in a future blog post). Regardless, my understanding of project management principals was getting far to rusty and I decided to read up.

The Fast Forward MBA in Project Management by Eric Verzuh, while likely an awful project management book for anyone serious about learning the deeper craft of project management was an excellent book for me. It runs through a brief history of the topic and then launches into terms, technique concepts and explains their purpose. Occasionally, the author is really talking to the reader as if they will be applying the techniques as a project manager. Given that this book is (appropriately) geared toward someone pursuing an MBA, it seems unlikely that a real (good) project manager would ever read this book.

Most of my colleagues said something like: "A book on project management... must be riveting." I responded that I was actually enjoying it, but around 85% of the way through the tide turned and from thereon out I wanted to gouge out my eyeballs. Miraculously, I pulled through and finished with vision in tact. I'd recommand this book to other business people that have the need to hire and/or manage project managers. The restraint learned by not gouging your eyeballs out at the end of the book can even be applied at times when working with a bad project manager.

The one thing that got me thinking in the book (and that's what I really want books to do) was the articulated different between projects and operations. It was clear and well laid out. It spoke to the intrinsic need for a fundamentally different management style for projects and for ongoing operations. While that itself may be obvious, it make me realize that many of the engagements we take on at OmniTI aren't just "a bit of both" they are both. I'm still collecting my deeper thoughts on this, but suffice it to say I have more understanding and respect for why it is so challenging to manage expectations when delivering fast-paced services offerings on the Internet.

Make the web a faster place. Pretty please.

nospam@example.com (Theo Schlossnagle) — Tue, 07 Jul 2009 10:36:59 -0400

Call to action! Make the web a faster place! Here's a short article on how I spent 45 minutes to improve user-perceived performance on our website. This is the low-hanging fruit of front-end web performance optimizations.

Most of you who read my blog are scalability or performance nuts. Most of you also cast the majority of your focus (like me) on the back-end infrastructure problems. Don't ignore the front-end when just a tiny bit of work can remove a huge amount of suck.

If everyone takes these steps, the web will be a more enjoyable place to visit.