Esoteric Curio - Damaged Bits

Surge is gonna kick ass.

nospam@example.com (Theo Schlossnagle) — Tue, 09 Aug 2011 23:48:22 -0400

I haven't blogged for a while because:

I have been travelling insanely. About 80k miles this year so far.
Hacking on Circonus and (subsequently) Reconnoiter.
Providing strategic and tactical guidance on some mind blowing projects for the truly awesome clientele we have at OmniTI.
Speaking at quite a number of conferences.
Attempting to participate more in some of the open source projects I can help.
... and planning for Surge.

The little timeout I am taking to write this note isn't sufficient to dive into the truly interesting things my readers usually expect. However, I'd be a jackass if I let you go any longer without knowing that Surge is going to be friggin' awesome this year. At the end of this year, if you look back and see that you missed attending Surge you'll know what regret truly is. Just sayin'.

Everyone is to blame for this continued expectation that such magic is possible.

nospam@example.com (Theo Schlossnagle) — Sun, 24 Apr 2011 20:29:59 -0400

My opinion is that the only reason the big enterprise storage vendors have gotten away with network block storage for the last decade is that they can afford to over-engineer the hell out of them and have the luxury of running enterprise workloads, which is a code phrase for “consolidated idle workloads.” When the going gets tough in enterprise storage systems, you do capacity planning and make sure your hot apps are on dedicated spindles, controllers, and network ports.

It was fantasy believing it was possible to pull off a centralized network block storage service in a multi-tenant cloud without any of the architecture shenanigans our enterprise brethren do and think that applications, databases, and business could depend on its being perfect. Honestly, we should have know better. We the applications developers asked what is perhaps the crappiest of all abstractions in computers to solve all of our availability problems for us. We asked for magic. Clearly, the vendor never should have made the promise of magic, but everyone is to blame for this continued expectation that such magic is possible.

- Mark Mayo of Joyent

This is from an interesting article by Mark Mayo over at Joyent with some excellent gems in it.

Surge 2011 CFP Extension

nospam@example.com (Theo Schlossnagle) — Tue, 05 Apr 2011 22:41:11 -0400

The CFPs have been rolling in for Surge 2011; these are exciting times. It does, however, appear that our description of what we're looking for has produced a different set of submissions that what I expected. I think it might help to better understand what sessions were like last year and, luckily, we'll be releasing all of the Surge 2010 video footage this week. I apologize for the poor audio quality, we intend to pull in A/V recording professionals this year.

We've received some great proposals! However, a surprising amount of them are presentations about products. This conference is about problems and their solutions. It is a conference for practitioners by practitioners. Blood... in the mud. I want to (as we did last year) share our struggles for better collective experience.

Emphasis: Accepted proposals will demonstrate real-life scalability challenges and creative solutions. We love case studies and learning from our mistakes.

As such, with this context, I'm extending the Surge 2011 CFP deadline to April 17th.

Top 5 "software to saas" culture shocks.

nospam@example.com (Theo Schlossnagle) — Tue, 01 Mar 2011 21:54:54 -0500

At OmniTI, I've been a part of writing a lot of open source software, my fair share of closed source software. Some of it has been shipped and some of it has been operated as a service. While it is possible (and quite useful) to take what one learns in one scenario and apply it to another, some things simply translate poorly.

I do a lot of consulting with traditional software companies that are looking to make a transition to the new world of SaaS. Again and again, I see the same "shockers" hit these companies during their metamorphosis. Read my top 5 things you need to know if you are going to change your business model from traditional software engineering to SaaS delivery.

Using Esper to manage real-time data.

nospam@example.com (Theo Schlossnagle) — Fri, 04 Feb 2011 14:01:10 -0500

I'm flying back from a wonderful event: Strata.

I gave a talk there called "Esperwhispering" that seemed to pique many people's interest. This is the stuff you do when a database just doesn't have the horsepower to answer your questions fast enough.

Esper is an excellent, open-source CEP tool. It's a shame its GPL, but hey... you can't win 'em all.

We use esper to power many things internally at OmniTI and our clients and Esper is the code CEP engine we use to make sure Circonus custsomers know when "things go wrong."

This presentation gives some insight into what it does and why one would use it. The live presentation, of course, had more information and live demos, but... you had to show up for that (or watch the live stream of the event).

Esperwhispering

View more presentations from postwait.

#ywahusty

nospam@example.com (Theo Schlossnagle) — Fri, 04 Feb 2011 10:41:44 -0500

Defining the term:

I recently used a term and was hit with a lot of out-of-band requests for explanation. It's a good one and excellent food for thought.

ywahusty (yuh-wuh-hus-tee): you will always have users smarter than you.

This basic concept is one of sound, pragmatic systems engineering that might appear to fly in the face of traditional product engineering... but doesn't.

In traditional product engineering, there is a goal to produce a product that is both accessible and useful to the largest subset of the predefined audience of the product. Basically, in software terms: you don't put too many knobs on the dashboard or you will have all sorts of operator errors. You work very hard to tune those knobs or have then auto-tune so that the product behaves correctly under normal operational uses (and of course, their edge conditions). This is just good product design.

We are faced with a problem: shit goes wrong. As an engineer for a software product, you know full well that shit will break. Those knobs will be set and/or auto-tuned and you'll be thinking: "what if?" The: "Wow, I've never seen this set of inputs. The min_wu_tang_threshold is 11... it might behave better at 42 right now... sure wish I could twist that knob that product management insisted I leave off right now." The truth is: the product you ship is not the product they receive. Not every knob you ship has to be in the "manual," it can be in the unsupported "ninja manual." You can absolutely instrument your application without sacrificing simple and intuitive use by the majority of your audience.

Bottom line? The knob should be there. My hat goes off to an engineering team that takes this to heart: "I am ten ninjas." And a second tip of the hat to the diaspora of Solaris kernel engineers that have allowed us to tweak (seemingly) insane operational constraints on production systems without applying patches or rebooting.

Appealing to the pragmatist.

I'll attempt to appeal to the pragmatist in you. You are equipped with more knowledge now than you were when you started reading this prose. This is a truth whether you agree with anything I've said simply because you have more information. Even you, with you current implementation knowledge would benefit from the ability to tweak, twist and manipulate the parameters of your operating product because it will allow you to test new ideas, verify assumptions and validate the design and implementation choices you've made with far greater ease.

As a side note, gdb and/or mdb aren't the "knobs" I'm talking about: think more along the lines of sysctl's on most UNIX implementations. These are (mostly) run-time tunable parameters -- and yes users can set them to values that will cause certain, and often immediate, combustion. (good) Kernel engineers know that price benefit of having these knobs far outweighs their liability.

Appealing to the experienced.

Now, we come full circle, for the honest engineers that can check their ego at the door. I have never in my life developed a piece of successful software that had a user-base that contained only users dumber than myself. The concept therein is absurd for any project with even moderate adoption. You engineer your systems to be both observable and operable because at the end of the day you will get schooled by (at least some of) your users.

Again... please make your software operable. Why? #ywahusty

Why haven't all my graphs been useful like this?

nospam@example.com (Theo Schlossnagle) — Thu, 20 Jan 2011 16:21:28 -0500

I don't know now many times I've tried to eyeball a graph from this week against a graph from last week. This is both painful and senseless as my eyeballs simply aren't that good. Enter "the right way to do it." Bliss.

Capacity Planning

nospam@example.com (Theo Schlossnagle) — Wed, 05 Jan 2011 19:33:46 -0500

I used to export data into Excel and do linear regressions on it... How I have no need. I am happier person and people like me more.

Here's how we made capacity planning easier for everyone.

Theo seeks aspiring database administrator

nospam@example.com (Theo Schlossnagle) — Tue, 28 Dec 2010 08:20:40 -0500

What does it mean to be a database administrator? It means more than just respecting data in a room full of engineers and analysts that do not -- it means bending them to your will and making them respect it, too. It means knowing when people's concept of sacrosanct data integrity and consistency can be thrown out the window and letting those people know that, this time, they can cope with inaccuracy or volatility. You think those things contradict? This job's not for you.

Do you think the world can be solved through relational algebra? I'm listening. Do you think the world should be solved through relational algebra? Don't apply here. Have you edited a database redo/write-ahead log in a hex editor to bootstrap a failed recovery? Now we're talking about reality. We live in a world where the probability of bit errors on disks rattles our cage by guaranteeing corrupt data and indexes when using traditional filesystems. Being responsible for data in this world is no simple task.

If you think you can scale single RDBMS to the sizes we see on a daily basis, I'll beat you with a stick.
If you think RDBMS is are passe and any problem can be appropriately solved with a NoSQL solution, I'll beat you with a stick.
If you think data is holy, I'll buy you a stick.
If you try to pipe my data through the wrong tool, I'll carve zeros and ones into my stick and then beat you with it.
If you think its okay to let someone who cares more about code than data roll out queries without your review, I'll make you beat them with a stick, then beat you with a stick while you clean up their bloody remains.

In a real database architect's role, you don't manage a database -- you manage data. You keep it secret, you keep it safe, you keep it clean, and you keep it available. From Oracle, to PostgreSQL, to MSSQL, to MySQL, to Cassandra, to Riak, to Hadoop, to Esper you will be responsible for the operational integrity of systems that touch data and turn it into useful outputs for people and businesses.

If you know relational databases and at least one of NoSQL systems or streaming databases (CEP/IEP), then you may have what it takes to be on the OmniTI data management team.

Web Performance Boot Camp

nospam@example.com (Theo Schlossnagle) — Tue, 02 Nov 2010 16:33:09 -0400

Slides are here, get 'em while they're hot.

Perform or Die

nospam@example.com (Theo Schlossnagle) — Fri, 29 Oct 2010 22:39:50 -0400

There's no excuse these days for crappy web performance. The low hanging fruit is just too low. It is trivial to make a web site that loads fast for users. Do it. If you don't know how, you should come to my Web Performance Boot Camp tutorial at ApacheCon. Make you site fast — no excuses.

Why JSON sucks.

nospam@example.com (Theo Schlossnagle) — Sun, 10 Oct 2010 09:19:17 -0400

JSON sucks. Don't get me wrong, I love the simplicity of it. It's simple, it's easy, it's portable, it's ubiquitous at this point. None of that means it doesn't suck. Outside of Javascript (hence the portability), JSON itself is limited to native types in the grammar:

null
object (like a hash)
array
string
integer (signed 32bit)
number (double)
boolean

Really? It's 2010 and we're all flocking to a grammar where we can only accurately represent integers up to 2³¹-1. WTF? Any new standard we adopted should certainly have had grammar specifications for: 1, 8, 16, 32, 64, 128 bit signed and unsigned integers, 32bit and 64bit IEEE floating point (float and double) as well as arbitrary precision real numbers. I'd much rather have a grammar that I struggle to translate into my native language data types (like, "hmmm, what am I supposed to do with a real in C?") than I would have a grammar in which I cannot precisely express myself.

Every time I need to (correctly) represent a large integer such as 4611686018427387900, I'm forced to do so in a string. It causes me to throw up in mouth a little. Everyone seems dead set on this, so I suppose I'll learn to cherish the flavor of bile.

Finding leaks on Solaris (w/o Valgrind)

nospam@example.com (Theo Schlossnagle) — Sat, 09 Oct 2010 16:04:18 -0400

Premise: I write a lot of C code. I run a lot of Solaris.

Sadness: One of my favorite tools ever made is Valgrind. Valgrind does not run on Solaris.

A lot of the C code I write is event-driven and as such (complicated) it is harder to write code and leaked memory is a common residual of this more complicated coding effort. Memory leaks suck. Most of the code I write is systems-level code and as such is exercised heavily and needs to run for months or years without restart. So, leaks more than suck.

Valgrind will tell you exactly where you're leaking (and it feels like it tells you why)... BFM.

On Solaris, we have a different option: libumem.

I compile almost all of my apps against libumem, which has the effect of replacing malloc/free/and friends with a more multi-processor scalable slab-allocator implementation than the default one in libc. A very useful feature in libumem is debugging and leak detection... how?

First, let's assume you've compiled your app with debugging symbols (-g in the compiler flags). Next, I'll assume you didn't have the foresight to link against libumem (-lumem on the linker line). We need to link in libumem and we need to turn on debugging. In the case of this example, our app is the Apache web server httpd.

LD_PRELOAD=libumem.so.1 UMEM_DEBUG=default /opt/apache22/bin/httpd

Now, just find the process ID of exampled (perhaps as easy as a pgrep exampled). It is leaking, so we'll use the process ID 666. To find the leaks (without killing or restarting), we can simply use the mdb umem helper ::findleaks on the running process.

echo "::findleaks -d " | mdb -p 666 > leak-profile.txt

Note you can also do this from a core... very nice.

The output has a summary and then individual profile records grouped by stack trace of allocation, like:

umem_alloc_16384 leak: 203 buffers, 16384 bytes each, 3325952 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         1d36b60          1d37000   70235c07811a50              197
                           65a468           617900                0
                 libumem.so.1`umem_cache_alloc_debug+0x152
                 libumem.so.1`umem_cache_alloc+0x1a2
                 libumem.so.1`umem_alloc+0xdb
                 libumem.so.1`malloc+0x59
                 libapr.so.0.2.12`allocator_alloc+0x466
                 libapr.so.0.2.12`apr_palloc+0xda
                 bio_filter_out_ctx_new+0x37
                 ssl_io_filter_init+0xb1
                 ssl_init_ssl_connection+0x2cd
                 ssl_hook_pre_connection+0x123
                 ap_run_pre_connection+0xc3
                 ap_process_connection+0x30
                 process_socket+0xa8
                 worker_thread+0x2d8
                 libapr.so.0.2.12`dummy_worker+0x30

mod_ssl... Why are you so mean to me?

What's your postgres search_path?

nospam@example.com (Theo Schlossnagle) — Sat, 09 Oct 2010 15:44:32 -0400

Hey you! PostgreSQL process running a query over there... Yeah you. What's your search path?

Hello? Why aren't you listening to me? Oh, just because you are busy running queries for someone else for hours means you don't have to take some time to answer my question? Apparently, that's a good enough excuse. You, yes you process ID 18883, need to respect my authority.

# echo '*postgres`namespace_search_path /s' | mdb -p 18883
0x9d5420:       noit_a29_n625680_noit, stratcon, public

I done told you.

Sorry. I didn't mean to get rough with you. You know I love you, right?

OmniTI's Surge. Year one. Done. Awesome.

nospam@example.com (Theo Schlossnagle) — Thu, 07 Oct 2010 11:00:00 -0400

This past Thursday and Friday (and much of Wednesday honestly), OmniTI went where it never went before: we ran a conference. Surge 2010 was, in fact, a unique conference despite its close relevance to others in the industry. This conference was assembled to answer one question: "How do I build systems that perform well and scale to meet demand?" That may seem like a simple question that someone has asked before, but I am an engineer and, as such, use an engineering definition of "system" and have a fearful respect for what "demand" can really be.

I'd like to shout out a huge CONGRATULATIONS to all of the OmniTI team involved in orchestrating this event. People stopped no fewer than fifty times during the two day event to tell me how professional they thought the event was: specifically the polish of the brand, the beautifully consistent quality and composure of the conference itself, the venue, the proficiency of the speakers and fact the the content presented was all on-topic and well oriented to, what I shall call, the Internet Architect Practitioner.

On Systems.

A system is a set of connected things or parts forming a complex whole and as such, it isn't "a computer." Systems, particularly complex systems, are not only fascinating, but also happen to be ubiquitous between all of us and every useful website on the planet. Designing a system isn't about writing a piece of software or running a computer, it is about so much more. In the context of Internet computing, it is about the connectedness of many components including logical, electrical, optical and even mechanical. Bryan Cantrill did a spectacular job enlightening the crowd to the fascinating intricacies of failures in complex systems.

In fact, in talking to Cantrill, he suggested that each talk should be required to have at least one real-world failure described. I personally believe that learning from another's success is far less effective than learning from their failures. Brian was describing his impression of Tom Daly's excellent talk on how Dyn leverages AnyCast and partway through he said to me: "Given the complexities of this system, I have to think when this thing goes south, it must go really south." True enough, Tom described a rather intricate cascading failure all summed up with two words: fascinating and educational.

On Demand.

Demand is, in the beginning and the end, the bedfellow of supply in the racy world of economics. The interesting part in Internet systems (as opposed to economics) is how the system reacts to conditions in which demand exceeds supply. In economics, inventory is exhausted and future prices rise. In our systems, resources are depleted and experience nose-dives. If this condition persists, we have a natural economic scenario where that demand for a crappy experience will not erode and general demand will fall bringing back balance. However, we can "just make more." We can add more resources and ensure that resources aren't depleted. And, most importantly, if you architect your systems well, you can do this both successfully and cost-efficiently. Demand on the Internet has a dark side in that while it is bounded, it is bounded by the population of planet Earth. Ouch.

Artur Bergman presented details on how Wikia serves its global audience both successfully and (very) cost-effectively. Artur Bergman never fails to make me think just a bit outside of the box I'd been thinking in; always refreshing and always useful. I've been struggling with how to accomplish the real-time variant of what Artur has done aggregating actual user-perceived performance; the problem isn't that hard, the trick is scaling it as a service. The idea that one can leverage Google Analytics to do this, for free, in hour-relative real-time is damn spiffy.

Learning.

I learned a tremendous amount at Surge 2010. My job, amongst others, is to keep abreast of the current technologies and methodologies used to power the largest websites in the world (top 500). I take that knowledge and extract or adapt those techniques and help companies in the top 10000 compete more effectively. I read academic papers, ACM and IEEE journals every week. I talk to colleagues. I read "trendy" tech outlets like hackernews and reddit. I gather as much knowledge as possible. But, what do I do with it?

My goal is to share that knowledge with my team at OmniTI. I've struggled with how exactly to do this in the past. I have to say that my most successful effort to date was Surge 2010. Almost everyone from OmniTI attended and many told me that they left Surge feeling educated and (more importantly) inspired by the truly enthralling industry in which we all work.