Esoteric Curio

Scalability and concessions

nospam@example.com (Theo Schlossnagle) — Wed, 02 Jul 2008 11:52:03 -0400

Oren Hurvitz has a great post about LinkedIn's architecture. It's well-written and well thought out. Their architecture has evolved on what appears to be a steady and safe path of improvement. It is well worth a read.

I would like to comment on something I see repeated again and again and is likely misinterpreted by young scalability architects. The statement of what you should expect to lose when you scale up/out. Oren writes:

The presentation ends with some tips about scaling. These are oldies but goodies:
Can’t use just one database. Use many databases, partitioned horizontally and vertically.
Because of partitioning, forget about referential integrity or cross-domain JOINs.
Forget about 100% data integrity.
At large scale, cost is a problem: hardware, databases, licenses, storage, power.
Once you’re large, spammers and data-scrapers come a-knocking.
Cache!
Use asynchronous flows.
Reporting and analytics are challenging; consider them up-front when designing the system.
Expect the system to fail.
Don’t underestimate your growth trajectory.

Now, I agree with much of that. The spammers comment should be revised to "Fraud happens and the bigger you are, the bigger the bullseye." Be aware and protect your assets. Everything from Cache! on down: hard and fast rules. The cost argument is odd. While it is completely correct, it's also rather obvious. If your business model ties audience size and site use to revenue (which it should), then the cost should simply scale sub-linearly w.r.t. revenues (i.e. no big deal). However, there are a few that remain on that list that should be cherished and the loss of them should pain you.

"[You] Can't use just one database" -- this is a conclusion you should arrive at after analysis. We have one client that supports 10 million users on a cluster of partitioned databases. We have another that supports 35 millions users on one database without issue and room for growth.

"Because of partitioning, forget about referential integrity or cross-domain JOINs." Think. Think hard. Think harder. Sometimes it is possible to partition in a fashion that allows for integrity. While I'm sure (or at least hope) that the LinkedIn guys had some sleepless nights making the decision to break foreign constraints, it isn't conveyed. You should absolutely have some sleepless nights over a decision like that. My bank supports many more users and transactions than LinkedIn -- and it damn well better have FKs and 100% integrity. So, while you still may partition in such a fashion that requires a loss of enforced integrity, the decision should be a heavy one.

"Forget about 100% data integrity." WTF? While I'm sure it was the end of the post and he was being smart, someone somewhere might actually take the advice to forget about data integrity. You never, ever, ever forget about it. We have some "one big database" architectures where data integrity has been an issue due to memory bit-flips (corrupt data on disk) -- it's a BFP (big f@#$ing problem) and we treat it that way. Sometimes you make an architectural decision that will make the loss of integrity much more probable (partitioning and losing FK constraints is a ripe example). It's still something that should be attended to with great attention and diligence. you should never forget about data integrity and always put forth the effort required to reach as close to 100% as possible. When you lose data integrity you end up with a big pile of shit in your database. I'll leave you with a rather crass metaphor:

There's an expectation that there is no shit on your living room floor. Don't shit in your living room. Don't let your dog shit in your living room. If you're a dog owner, you know your dog could have an accident. You bought the dog. You chose to increase the probability of finding shit in your living room. Don't ignore it or forget it. Clean up the shit when it happens. If you get suddenly ill while playing your Wii naked and shit on your living room floor (be it probable or improbable)... respect yourself -- clean it up. Never forget the goal: a 100% shit-free living room.

Reconnoiter and another platform

nospam@example.com (Theo Schlossnagle) — Fri, 27 Jun 2008 17:27:27 -0400

Reconnoiter is coming along. Unlike most open source project, I tend not to talk about mine until their are really useful to people. Over the last year, I've adopted the unhealthy attitude that useful means "shiny front-end." So, I'm blogging to break that attitude and talk a bit about project that doesn't have a shiny front-end... yet.

Reconnoiter is built out of years of frustration using tools like RRDTOOL, Munin, Cacti, ZenOSS, Nagios, etc. etc. I have a lot of problems with these tools. First, they are not efficient. I need a powerful machine to monitor a mere 10k services. And it actually gets to be an engineering challenge to monitor 100k services with these tools. Also, the graphs are about 10 years old with respect to design and usability. I want something new, something fresh, and something that doesn't need a damn web UI to configure. Several people have asked, why are you reinventing the wheel? Why don't you just improve an existing product? My answer is that I want a well-thought-out product foundation so that I can trust all the bits. I want reponsibilities decoupled at the right spots. I want data in a form that the world can query and run reports the likes of which I have not concieved. I don't want the load on my monitoring machines to be 8. I want my monitoring system to check services and metrics when it planned to, not several minutes (or event 2 seconds) after it told me it would. Simply put, I expect it to work well, all the time. And, of course, I want it to work how I would expect it to work.

Reconnoiter was born out of the need to monitor the internals of many disconnected data centers with between 10 and 1000 machines in each facility. Monitoring can mean a lot of things, here I consider it to be the collection of metrics and awareness of their availability. In and of itself, monitoring is pretty useless, but it is the foundation for two critical pursuits in Internet infrastructure and business management: fault detection and trending.

Fault detection is as simple as understanding when something has faulted. However, knowing something is broken is easier than knowing something is about to break. Is it better to know that your machine just crashed because the chip slagged to the motherboard, or that the temperatures in rack 043 are rising unexpectedly? Answer: both, but I hope I only learn the latter and not the former. Truly, there are too many things to monitor... hundred or thousands of metrics on each piece of equipment. I can't reasonable go in and configure good/bad thresholds on each one. I want anomaly detection. I want a system that I can say: "this looks right, tell me when it stops looking right." That, to me, is a much need companion to tradition fault detection.

To me, trending is much more than drawing graphs... it is about intelligent data correlation, regression analysis/curve fitting and looking into the past to see how much you fucked up getting where you are now -- in the vain hope that you learn from your mistakes and plan better next time.

Reconnoiter is an attempt to build these things. Building a system requires starting with pain (need), solid structure and plumbing (good engineering). So, reconnoiter is underway. And this post is in mid-step:

It started on OpenBSD, and added support for FreeBSD, Mac OS X, Linux.

As of changeset [292], we have Solaris/OpenSolaris support.

We have a pretty nice front-end for trending under construction, but it isn't there yet. We'll have numeric data combined with textual "event" data on the same graphs. All that convenient stuff. Here's the rather plain-Jane graph you get now (because some people won't even read a post if it doesn't have a pretty graph):

Honestly, I don't know what the value of this post is, but people around here keep telling me that people should be aware of an open-source tool like this, even if it isn't finished (read: usable) yet. I say it isn't usable yet, but on our development instances here, we monitor 2892 production metrics across two data centers and the load never peaks past 0.10. I'm pretty excited about where this is going. Honestly, my favorite part right now is that I can configure and control the noitd checking nodes via a telnet console and it acts as if it is a piece of network equipment rather than an "application" -- as it should be IMHO.

Dissecting today's Internet traffic spikes

nospam@example.com (Theo Schlossnagle) — Tue, 20 May 2008 13:56:59 -0400

Today's Internet has changed quite a bit from the Internet I used to know. The Internet has always been successful because of net neutrality. What's net neutrality? It's complicated, but essentially it means that anyone anywhere can publish with equal rights. These aren't the kind of rights people usually talk about... I'm not speaking of freedom of speech. Instead, I'm talking about content being simply bits. It doesn't matter if it comes from CNN or this blog, you as a reader can download the bits that make up the pages you see without bias or preferential treatment. This makes it darn easy to be a publisher and leads to a fabulous ecosystem with an overwhelming amount of varied content. However, with more content it is easy to recognize that much of it is utter trash. Yes. Yes. I know that one man's trash is another man's treasure. However, it presents opportunities for sites that help you navigate the wasteland.

Many popular sites today are popular because they link to articles and news items and photographs and movies all over the Internet; they are "interest aggregation services." And while the Internet has (for now) a decent preservation of net neutrality when it comes to simple web content, not all publishers are on equal footing. Not long ago, anyone could run a server anywhere (their basement) with DSL or cable or (gasp) dial-up -- now, the challenge is coping with unexpected attention.

Years ago, the site slashdot coined a term "slashdotted" which meant that a site received so much sudden traffic that service degraded beyond an acceptable point and the site was effectively unavailable. This often happened to sites that were at the end of small pipes (DSL, T1, etc.) and occasionally (though rarely) due to bad engineering. While slashdot might have coined the term, they simply don't have the viewership numbers that other large sites today have.

At the $DAYJOB, I work on sites that aren't on the end of T1 lines. Sites with gigabits or tens of gigabits of connectivity. Sites with 50 millions users. Sites powered by thousands of machines. I also work on sites that service millions of people from just a handful of machines (efficiency certainly has its advantages sometimes). I find it particularly interesting that already popular sites (with significant baseline bandwidth) are seeing these unexpected surges. For a long time, my blog has been on this same machine which is a vhost for several other web sites. I've had traffic spikes from places like slashdot, reddit, digg, etc. And, no surprise, I couldn't actually see the bandwidth jump on the graphs... 10Mbits to 11Mbs? That's not a spike.

Things are changing. Sites like Digg are becoming ever more popular and people are drawn to them as a means of sifting the waste of the Internet. This means as more people rely on Digg and Reddit and other similar sites, the number of unexpected viewers of your content can rise more sharply.

What does all of this mean? It means that the old rule of thumb that your infrastructure should see 70% resource utilization at peak is starting to falter. The typical trends used to look like this (this is last week's graph from a retail client with a user base of 3 million):

We see a nice peak, a nice valley. Thursday afternoon, we see a nice traffic spike. Well, this used to be what I called a traffic spike. Now, different services have different spike signatures. It resembles traffic model of classic Internet advertising, except that there is genuine interest and thus dramatically higher conversion rates. It's a simple combination of placement, frequency and exposure. Because content, unlike ad banners, exists for an extended period of time (sometimes forever), the frequency is very high. Digg and Reddit have excellent placement with very little exposure (things move out quickly). A site like CNN or NYTimes usually provides mediocre placement (unless you are on the front page) and excellent exposure.

Lately, I see more sudden eyeballs and what used to be an established trend seems to fall into a more chaotic pattern that is the aggregate of different spike signatures around a smooth curve. This graph is from two consecutive days where we have a beautiful comparison of a relatively uneventful day followed by long-exposure spike (nytimes.com) compounded by a short-exposure spike (digg.com):

The disturbing part is that this occurs even on larger sites now due to the sheer magnitude of eyeballs looking at today's already popular sites. Long story short, this makes planning a real bitch.

And the interesting thing is perspective on what is large... People think Digg is popular -- it is. The New York Times is too, as is CNN and most other major news networks -- if they link to your site, you can expect to see a dramatic and very sudden increase in traffic. And this is just in the United States (and some other English speaking countries)... there are others... and they're kinda big.

What isn't entirely obvious in the above graphs? These spikes happen inside 60 seconds. The idea of provisioning more servers (virtual or not) is unrealistic. Even in a cloud computing system, getting new system images up and integrated in 60 seconds is pushing the envelope and that would assume a zero second response time. This means it is about time to adjust what our systems architecture should support. The old rule of 70% utilization accommodating an unexpected 40% increase in traffic is unraveling. At least eight times in the past month, we've experienced from 100% to 1000% sudden increases in traffic across many of our clients.

I talk about scalability a lot. It's my job. It's my passion. I regularly emphasize that scalability and performance are truly different beasts. One key to scalability is that a "systems design" scales. Architectures are built to be able to scale, they are not built "at scale." It's just too expensive to build a system to serve a billion people (until you have a billion people). It's cheap to design a system to serve a billion people. Once you have a billion people accessing your site, you can likely justify executing on your design. Google is successful for this reason: their ideas scale and they can build into them as demand rises. On the flip side, traffic anomalies in the form of spikes are unexpected (by their definition) and scaling a system out to meet the unexpected demand is almost unreasonable. I would even argue that it is more of a performance-centric issue. I want every asset I serve to be as cheap to serve as possible allowing me to handle larger and larger spikes.

The reason I find all of this stuff interesting is that understanding performance and scalability, understanding the principles of scalable systems design and having sound and efficient processes for handling performance issues is becoming crucial for sites regardless of their size. This takes insight and practice and it reminds me of Knuth's famous saying:

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

That's all well and good, but which 97% of the time? My response to Knuth's statement (with which I completely agree) is:

Understanding what is and isn't "premature" is what separates senior engineers from junior engineers.

Let's add perspective on the word "sudden." Most network monitoring systems poll SNMP devices (like switches, load-balancers, and hosts) once every five minutes (we do this every 30 seconds in some environments). Some people say, "my site scales! bring it on." We see these spikes happen inside 60 seconds and they occasionally induce a ten-fold increase over trended peaks. Often times, this spike can be well underway for several minutes before your graphing tools even pick up on it. Then, before you have time to analyze, diagnos and remediate... poof... it's gone. Be careful what you wish for.

This, in many ways, is like a tornado. Our ability to predict them sucks. Our responses are crude and they are quite damaging. However, predicting these Internet traffic events isn't even possible -- there are no building weather patterns or early warning signs. Instead we are forced to focus on different techniques for stability and safety. The idea of a DoS, a DDoS or the sometimes similar signature of a sudden popularity spike doesn't increase my heart rate anymore -- it's just another day on the job. However, I thought I'd share the four guidelines that I believe are key to my sanity in these situations:

Be Alert: build automated systems to detect and pinpoint the cause of these issues quickly (in less than 60 seconds).
Be Prepared: understand the bottlenecks of your service systemically. Understanding your site inside and out. Contemplate how you would respond if a specific feature or set of features on your site were to get "suddenly popular."
Perform Triage: understand the importance of the various services that make up your site. If you find yourself in a position to sacrifice one part to ensure continued service of another, you should already know their relative importance and not hesitate in the decision.
Be Calm: any action that is not analytically driven is a waste of time and energy. be quick, not rash.

Back to those other countries... Enter China and their recently lessened censorship and we have a looming tidal wave for smaller sites that achieve sudden popularity. Spikes of several hundred megabits per second are difficult to account for when your normal trend is around twenty megabits per second. The following graph is traffic induced from a link from a popular foreign news site (that I can't read). I call it: "ouch:"

BWPUG Meetup Reminder

nospam@example.com (Theo Schlossnagle) — Tue, 13 May 2008 20:49:56 -0400

Hi all!

Just a friendly reminder that we'll be having our first meetup tomorrow as planned. I thought as a good kick-off we could all collaboratively share what we do with PostgreSQL. We'll start off with a whirlwind tour of how OmniTI uses PotsgreSQL, taking a brief look at ZFS, DTrace and large datasets. After that I think it would be good to get to know each other -- maybe we'll hit a local pub afterwards!

I look forward to seeing you there!

Meetup starts at 6:30pm
7070 Samuel Morse Dr. Ste 150
Columbia, MD 21046

If you have issues getting in the building, ring me on my cell -- it will be posted on the doors.

Best regards,

Theo

OSCON 2008: And now for something completely different.

nospam@example.com (Theo Schlossnagle) — Sun, 27 Apr 2008 22:02:28 -0400

I just registered for OSCON. They say I should advertise that I am a speaker. Here goes.

For the last several years, I've presented multiple talks at the O'Reilly Open Source Conference. My Scalable Internet Architectures talk has been quite popular and drawn large crowds. It is an interesting talk as it doesn't really change with time. As I say, "if principles of good engineering changed frequently, I'd never drive on bridges." The talk is about sound engineering approaches to building really large consumer-facing websites. Almost all of it is open-source centric, which is why it fits so well at OSCON. While my Scalable talk was not accepted this year, I've got another talk lined up that will rock your world.

I am quite excited that my other proposal was accepted. This year I will be giving a session about using DTrace to perform "full-stack" introspection.

Using DTrace we will deep dive into the amazingly cool questions one can ask. Is my application really hitting disk? If so, what line of code is causing it? My process is being descheduled by the kernel, why? I have 100 Apache process and some randomly segfault, how do I get a stack trace when that happens? The app I am running doesn’t have the right debugging output, I need to know more!

DTrace is an oracle. The value of the answers depends on the quality of the questions. Learn to ask good question and prepare to be amazed at the possibilities.

I've given a variation on this presentation at a few places now (both internal to OmniTI and external) and had really positive feedback. I'll be taking these prior presentations and polishing them up for a 45 minute escapade that will open your eyes to new possibilities. DTrace is an amazing tool and once you get used to it, you can really take it for granted. I do. When people watch the presentation and say "by the power of Greyskull," I know I've made my point.

Come to OSCON. Immerse yourself in technology.

ZFS. Respect.

nospam@example.com (Theo Schlossnagle) — Tue, 22 Apr 2008 23:16:32 -0400

Today someone asked me: "You speak about ZFS a lot. I know other people that talk about the latest filesystems with praise, but generally speaking they just don't have much to offer. Is ZFS that different?"

My answer is "yes." But, of course, I can't leave it at that. I'm not going to make a performance argument -- ZFS is fast in some cases and slow in others -- just like everything else. I think one of the things we've seen in the last 10 years is that everyone felt the need to come out with their own filesystem -- at least on Linux. So, you have to as yourself why. My personal opinion is that filesystems on Linux suck.

Most filesystems on the market support snapshots. No open source filesystems on Linux (that I'm aware of) support snapshots. Of course, you can use LVM to do block-level snapshots. First off, that's a pain in the ass w.r.t. storage provisioning. Other systems make the process of allocating and managing snapshots "not my problem." (simple and easy). Let's be frank, ext2 and ext3 are nothing to write home about. reiserfs, xfs, jfs, the list goes on and on.

There are a few closed-source filesystems that are really nice. Specifically Veritas Filesystem (VxFS) and its excellent layered volume manager VxVM which appears to have heavily inspired geom on FreeBSD. DEC thought it was so cool that they pulled it white-label into Tru64. Respect.

So, what makes ZFS so different? ZFS is a disruptive technology as it abolishes the sacred line in the sand between block devices, volume management and filesystems. This means it just make storage management easy. When I say easy... I mean easy.

So you want more space? Add more disks. Want to move from from failing disks to replacements? Tell zfs to add the new ones and tell it to remove the old ones. Read that report by Google about disk errors? ZFS checksums all data. My personal experience says checksums are good. Snapshots? Sure snapshot to your heart's content. We snapshot some systems hourly and never ever delete the old ones. Snapshots are really cool, but what if you could rollback to a snapshot? zfs rollback. What if you wanted to make a read/write copy of the fileystem or an old snapshot? zfs clone. You want to store a lot of raw data? zfs has built-in compression. Oh, and it is open-source.

Simply put. ZFS. Respect.

Starting the Baltimore/Washington PostgreSQL User Group

nospam@example.com (Theo Schlossnagle) — Mon, 14 Apr 2008 13:49:42 -0400

On the second Wednesday of every month, the Baltimore/Washington PostgreSQL User Group will meet at 7070 Samuel Morse Drive, Ste 150 in Columbia, Maryland. Meetings start at 6:30pm and go until around 8:30pm. I am pretty excited about this and pleased to offer up OmniTI's facilities for this. I'm excited about the opportunity to share what I've learned, educate and grow the PostgreSQL community and learn from others in it. This is going to be "good stuff."

Our first meeting will be held on May 14th, 2008. Mark your calendars. Also, subscribe to the mailing list at: bwpug@postgresql.org.

Probing for Success

nospam@example.com (Theo Schlossnagle) — Sun, 13 Apr 2008 15:08:22 -0400

I recently attended dtrace.conf(08), which was a blast, but I left that conference with a single thought and it has been reinforced since. Everything should be dtrace enabled. While it is true that using DTrace you can introspect just about everything in the system, the pid provider (used to trace inside user-space applications) requires the user to to know the code of the application. A full system "in-flight" has too many different apps running for me to keep all of their code-bases in my head. The kernel and one or two apps is about my limit. Also, the pid provider is somewhat limited in that it makes watching a lot of processes on that level intractable. So, what's the solution? SDTs for user applications or USDTs.

USDTs allow application authors to put specific probe points in their software that provide three compelling advantages over process-level tracing with the pid provider: (1) they should be carefully placed and named to be accessible to a user who does not know the application code base, its data structures, or even C for that matter, (2) they boast a very low barrier to entry, and (3) they work system-wide as expected.

Sounds easy, right? Well, adding the code to an existing project is "retarded simple." After a few lines of patching in the build process (usually a makefile), new probes can be added at about two lines of code per probe (one line for the probe and one line for the prototype). So, it is easy. But, why isn't every application pimped-out with DTrace probes? Linux. Linux doesn't have DTrace and as such, I think there is a lot of resistance to add the probes to software who's primary development base (and target) is Linux. I don't think they are against it, but the "what's in it for me?" question comes up and acts as an obstacle. This is the case with any "cool new technology" that's not mainstream.

The real challenge is that each open source project (to which we would add probes) has its own culture and process for proposing changes, submitting patches, negotiating for inclusion, etc. In a lot of ways, in order to effect change, we have to take the role of a package distributor and with enough impetus, we'll see the upstream pull our patches on their schedule without much cooperation from us (of course, we're also happy to cooperate).

I'm not going into the details of why DTrace is hands down better than peanut butter and jelly -- you need to see it in action to truly respect it. However, with about two hours worth of work in PostgreSQL, I exposed probes in some parts of PostgreSQL that are otherwise hard to inspect. I instrumented some XLog operations, checkpoints, "exec" nodes in the executor, buffer syncing, LRU operations (that drive the CLOG, SUBTRANS and MultiXact system), the autovacuum system and SQL executions from clients.

Look ma! I can watch my checkpoints in real-time:

CheckPoint initated...
  CLOG                                                              1
  SUBTRANS                                                          1
  Buffers                                                        1430
CheckPoint complete: elapsed 92355ms

CheckPoint initated...
  CLOG                                                              1
  SUBTRANS                                                          1
  Buffers                                                         911
CheckPoint complete: elapsed 55933ms

55 seconds! Is that evenly distributed?

  buffer writes                                     
           value  ------------- Distribution ------------- count    
             < 0 |                                         0        
               0 |@@                                       5        
               1 |@@                                       5        
               2 |@@                                       5        
               3 |@@                                       5        
               4 |@                                        4        
               5 |@@                                       5        
               6 |@@                                       5        
               7 |@@                                       5        
               8 |@                                        4        
               9 |@@                                       5        
              10 |@@                                       5        
              11 |@@                                       5        
              12 |@                                        4        
              13 |@@                                       5        
              14 |@@                                       5        
              15 |@@                                       5        
              16 |@                                        4        
              17 |@@                                       5        
              18 |@@                                       5        
              19 |@@                                       5        
              20 |@                                        4        
              21 |@@                                       5        
              22 |@@                                       5        
              23 |@@                                       5        
              24 |@@                                       5        
              25 |@                                        4        
              26 |                                         1        
              27 |                                         0        

CheckPoint complete: elapsed 26085ms

Well, it only took 26 seconds that time. It turns out that in PostgreSQL 8.3 the regular checkpoints attempt to spread out the buffer writes and we can witness it working well. Pretty neat.

With all these probes, the questions you can ask can be quite interesting. Particularly, how many xlog inserts did a query induce, or how many buffers did a query dirty. All as simple as a few lines of DTrace script now... in production... no configuration changes required.

Being able to ask systemic questions is of fundamental importance when troubleshooting problems. DTrace is built around the concept that things should be looked at systemically. However, to support that, we need applications to assist by exposing USDT probes in the "right places." We've started a project here at OmniTI called Project DTrace that aims to do just that. It's open, please join in!

PostgreSQL: Looking under the hood with Solaris

nospam@example.com (Theo Schlossnagle) — Sun, 30 Mar 2008 09:21:05 -0400

For those interested, here is my slide stack from PostgreSQL Conference East '08. I think the title of the talk was "PostgreSQL: Looking under the hood with Solaris."

The presentation was 90 minutes long and had lots of shell-based show-and-tell. Obviously that stuff isn't available in the slides. I think it went over quite well. The audience was small, but hopefully people took away the a lasting impression of what DTrace has to offer and at least one person had the response: "By the power of Greyskull." Regardless, enjoy the slides!

PostgreSQL Community

nospam@example.com (Theo Schlossnagle) — Sat, 29 Mar 2008 10:42:08 -0400

I just attended the Keynote by Joshua Drake from Command Prompt. There are a lot of good movements on the operational organization of the PostgreSQL community. I think his vision of the community is more aggressive and structured than many are prepared for, but in a community as large as the PostgreSQL community it is very good to have someone pushing the envelope and attempting to apply a vision. I don't want to go as far as Josh wants to do, but we'll wind up part of the way there and that "just perfect."

He used a lot of geek marketing terms going so far as to use the term "Db 2.0." I'll add a few comments and marketing terms to my commentary. Josh said we need to stop following and start leading; stop looking at features in Oracle as a future feature map. What Josh means here is that we need to be disruptive. We need to implement things no one else has. I think we need both.

Josh said he wanted everyone using PostgreSQL and not "the Dolphin." I have to say that I think his statement was too strong to match my opinion. I like PostgreSQL. I believe it is the right tool for the job more often than not. However, there is a "not." In fact, there are a lot of "not"s. There are many requirements that, when explored, map better to the offerings of an Oracle, a DB2, or, dare I say, even a MySQL. MySQL is the "right tool for the job" for many requirements. There is room for more than one database. In fact, there is room for all databases. Good business and engineering practices should always define the process of evaluating technology appropriateness. One good business practice is placing only technology that can be well supported by your existing (or easily accessible) engineering talent. This practice should never be confused with evangelism and zeal -- we all have these character traits, but good engineers and managers shouldn't use them as a part of defining the appropriateness of a solution.

Josh's primary goal seems to be to grow the community which will better legitimizes PostgreSQL. The one thing I took away from this is that I should make sure PostgreSQL is a more common topic of discussion. I think we should start a Baltimore/Washington PostgreSQL User Group; OmniTI will provide facilities, coordination and even food and drink as long as it is under thirty people.

I'm sitting at PostgreSQL Conference. I have a snazzy slide set. I have forgotten my Mac 15" VGA adapter. @#$^ *@#$*%! I'm sure I'll be able to borrow an adapter from someone, but I must note that this conference has a much smaller Mac to PC laptop ratio than any other other conference I've been to in the last two years. (a.k.a. a large group in denial).

PostgreSQL Conference East '08. Bring it... Yeah.

nospam@example.com (Theo Schlossnagle) — Wed, 26 Mar 2008 21:50:20 -0400

I was just surfing Planet PostgreSQL and read Selena Deckelmann's blog post that said the 2008 PostgreSQL Conference East is this Saturday and Sunday. My first response was, "WTF? I though it was like two weeks away and on a Thursday and Friday." My second response was, "I'm speaking at that, I should go." My third though? You should go too.

So... I, like any professional speaker, finished my slides on time and turned them in several weeks ago. I have a polished presentation that's going to kick major ass. If you are in the area (Baltimore/Washington), you should definitely come down and partake in the festivities that surround the most advanced open-source database in the universe. I might even through in a few Hot Fuzz quotes during my presentation.

Talking w/ Sun

nospam@example.com (Theo Schlossnagle) — Sun, 16 Mar 2008 23:21:55 -0400

New acquaintances walk away from their first conversation with me and either think that I am in love with a particular vendor or technology or they think I truly hate all technology. Both are true in some fashion.

The fact that I have an OpenSolaris feed on my blog might indicate that I'm a fan of Sun. The truth is I am and I am not. As is true of any large organization, it's really tough to be enamored with all of it. I am a huge fan of Solaris 10 and Sun's initiative to support strict ABI compatibility for stable interfaces, and I'm downright giddy about their ZFS and DTrace technologies. I think Zones/Containers are cool and I think their engineering team has some brilliant shining stars and is on the whole smarter than average. Yet, the OpenSolaris community is challenged in a lot of ways due to the corporate involvement by Sun that leaves me with a funny taste in my mouth. I'm luke-warm about Java and feel like their hardware initiative is going down-hill with bad quality problems on some of their new offerings compared to their spectacularly rock-solid history (sans the E4500).

Recently, I did an interview with Mark Thacker of Sun about our use of Solaris for their Solaris Podcast series. We've had some bad experiences here and there, but all-in-all it has been a win. DTrace has been a god-send and ZFS has saved my bacon several times. Anyone who's talked to me knows that I'm brutally honest and appreciate those that return the favor. I'll look at solution and I tell you what you did wrong. I don't tell you what you did correctly... after all it was all supposed to be done correctly. So, upon listening to this Sun podcast, several of my colleagues said: "that had to have been edited." As marketing people usually do, they attempt to limit the negative exposure as much as possible -- most notably, they removed a section about lack of tight integration between ZFS and Zones which has made for some very painful upgrade paths. We have marketing here at OmniTI too, I know the drill. All-in-all, I think the interview went rather well and fairly represents the benefits we've realized by deploying Solaris 10.

dtrace.conf(08)

nospam@example.com (Theo Schlossnagle) — Fri, 14 Mar 2008 21:28:49 -0400

As many people already know, I'm a big fan of DTrace. Well, today I attended dtrace.conf(08) -- the first (un)conference revolving around planet DTrace. It was awesome. Many people who know me well have heard me say, "my good days are when I'm the dumbest person in the room." That's not to be confused with "I like having bad days." Instead, I like to be at my best and still struggle to keep up. Here at dtrace.conf(08), the people here are damn smart -- a bunch of higher level thinking.

I gave a demo of the PostgreSQL stuff we do using dtrace. I put back my SDT postgres probes into our internal packaging systems -- I hope that I can get those back into PostgreSQL soon... I talked very briefly about the ZFS magic we've experienced, but as the conference was focused intensely on DTrace, I saved most of that for PostgreSQL Conference East 08.

As usual, I particularly enjoyed Bryan Cantril's infectious excitement and pungent humor. I'd like to thank Sun and the DTrace Team for running the conference (as well as the sponsors). I really appreciate the effort. DTrace has empowered OmniTI to make our customers more successful. Hands-down awesome technology.

I've said it before and I'll say it again. ZFS and DTrace are the most impressive operating system advances I've seen this decade.

A job, a mission, a career: all without a path or a name.

nospam@example.com (Theo Schlossnagle) — Wed, 16 Jan 2008 09:39:02 -0500

I'm sitting in the SFO airport waiting to sit on a plane for 6 hours to fly home from the O'Reilly Velocity Summit. Was it worth it? You betcha.

What is this Velocity Summit thing? It was a bunch of web architects from highly trafficked sites sitting around talkin' smack. It was operated in Foo style. However, one thing that made me really appreciate this meet-up was the lack of self-importance displayed by attendees. Everyone was just there to talk -- not to make people understand how much they knew. We were talking about The O'Reilly Velocity Web Performance and Operations Conference: what it should be and why.

Two things that I walked away with were (1) a realization of the lack of a career path for people who do what we do (no standard titles, no standard roles and responsibilities and certainly a lack of sex appeal) and (2) a clear lack of terminology for the technology requirements that are so common in these environments. Terminology is easy, in my opinion -- you just argue until someone wins. Of course, arguing is a hobby of mine, so I have bias. On the other hand, defining a career path that is an industry accepted path is hard.

Job Title

The term Web Operations was used a lot during this event. While it isn't awful, I really don't like this term. The hard part is that the captains, superstars, or heroes in these roles are multidisciplinary experts. They have a deep understanding of networks, routing, switching, firewalls, load-balancing, high availability, disaster recovery, TCP & UDP services, NOC management, hardware specifications, several different flavors of UNIX, several web server technologies, caching technologies, several databases, storage infrastructure, cryptography, algorithms, trending and capacity planning. The issue: how can we expect to find good candidates that have fluency in all of those technologies? In the traditional enterprise, you have architects which are broad and shallow and their team of experts which are focused and deep. However, in the expectation is that your "web operations" engineer be both broad and deep: fix your gigabit switch, optimize your MySQL database and guide the overall architecture design to meet scalability requirements.

I struggle with this. Not everyone can be a superstar. More importantly, no one can really start as a superstar. If we use an apprentice model (which is common in industries without institutional support) we limit the total number of able workers in this field. So, how do we (re)define the requirements for a junior web operations person?

We have to have a plan for hiring on people and progressing them through a career path to make this a legitimate discipline. One person said they just hire people that they think are agile -- "If I tell them to know IOS well enough to configure a router and troubleshoot a problem, I expect them to show up tomorrow with a basic understanding of IOS and ready to start typing in commands at a console." I agree this sort of "no boundaries" attitude is required for the job, but where do you start?

Another person mentioned that the reason for the lack of sex appeal in the position was due to popular attitude. Many people apply for development positions and "don't quite make the cut" and are instead offered system administration positions. I personally don't subscribe to this philosophy and we certainly don't operate like that at OmniTI, but I've see it in other companies -- I hope it is not prevalent.

Basically, this is one of the few positions in the organization that has no boundaries of responsibility. If something breaks, it is your problem. Why isn't this the case throughout the organization -- why is it that even the most junior of developers doesn't wake up to fix their code when it breaks and causes service degradation in the middle of the night? It's uncommon that this level of responsibility is expected of developers, while it is a quite common expectation of the operations crew.

Circling back, I really don't like the term "web ops." I realize it is not far off, but it isn't sexy. Google has a few different roles with this level of responsibility. One I like is called: "Site Reliability Engineer." However, I'd like a set of job titles and a progression through them that makes this an appealing career path for young, ambitious geeks.

In order to define these roles, we should think about what they are responsible for. In our organization I see this as a few things:

Junior

On the junior level, they are responsible for learning. They are responsible for deploying new services and documenting such deployments. They are responsible for instrumenting deployments to make sure that faults are detected and trending is possible.

Mid-level

On the mid-level, they are responsible for all of the above, and more. Effective and complete troubleshooting of failures. Making sense of trending information. Understanding work loads that exist. Tuning systems to better accommodate current workloads and proactive tuning to handle known future workloads. One of the key differences between mid-level and junior is the ability to correctly prioritize remediation of issues during incident response. Staying calm, collected and executing with clarity of thought during an emergency.

What does "complete troubleshooting" mean? I mean troubleshooting without boundaries. I want no shyness in cracking open developer code and telling them what they did wrong and why. Finger pointing at people simply doesn't work, you have to point your finger at implementation problems, not people. To do that requires the skill to track a performance problem or reliability issue down to a specific line of code or approach.

Senior

On the senior side, technology research and selection is a must. Incorporating new technologies in the architecture to improve availability and reduce costs. Constant analysis of systems to improve efficiency. Capacity planning to understand growth well enough to ensure provisioning and deployment outpace need. Donald Knuth long said that premature optimization is the root of all evil; I've long said that the ability to accurately determine what is premature separates senior from junior.

One of the core responsibilities that must be handled on all levels is assessing the appropriateness of the technologies at hand. At the highest level, the "Web Architect." one must ensure that technology selection as well as development and deployment strategy match the business need. This is "hard."

Above and Beyond

This is a special role. In a lot of ways, this role isn't for failed developers, it is for developers/engineers that have outpaced their career path. One that has a deep understanding of how things work: "a complete systemic view of general site architecture." However, they want more responsibility, they want to make sure that all of it works all of the time: the app, the stack, the hardware, the network. Whatever technology the business needs, it must work, it must performs and it must be able to meet demand. Lastly, in their heart of hearts, they must believe that all problems are equal in their need for resolution and problem prioritization is dictated by business impact and not by flights of fancy (how cool or interesting the problem is).

It's an impossible job requirement: "Knows everything about all technologies deployed in Internet architectures." While no one fills this req., what I want is someone who's career goal is to find out how close they can get. You up to the challenge?

MySQL: all their base are belong to others.

nospam@example.com (Theo Schlossnagle) — Fri, 21 Dec 2007 13:24:18 -0500

MySQL: all their base are belong to others.

Unless you are running MyISAM (stop reading and switch now before you lose data)... the backend you run in MySQL has a product roadmap controlled by a competitor. Think about that.

To be clear, I think this does not lessen MySQL's value today. What it does is make me entirely unsure of MySQL's future value. What's next? Luckily, I'm patients, so I'll just spectate.