Esoteric Curio - OpenSolaris

OmniOS

nospam@example.com (Theo Schlossnagle) — Thu, 12 Apr 2012 15:04:01 -0400

And this is it ... OmniOS.

The desktop and server: oil and water.

nospam@example.com (Theo Schlossnagle) — Fri, 23 Mar 2012 21:48:20 -0400

I've been using these computer things for a while. I've written what is now over 100k lines of production C code and many thousands of lines of code in a variety of other languages. I've seen my software run and I've run other people software. One thing they all have in common is their propensity to break under unforeseen circumstances. Shit happens.

On my laptop, I don't care much. I want nice, I want convenient, I want new and pretty and productive. I'm willing to tolerate a nominal amount of instability for those desires. I have to reboot my Macbook Air at least once every month to accommodate general software failures and security updates. There is, after all, a lot of crazy software on my workstation. (yes I call my laptop a work station as it is the station at which I work -- how many of your do the same?)

Somehow, somewhere some engineer and their cult following decided that they wanted to make a datacenter operating systems be more desktop friendly and a product manager somewhere and their hefty company decided they wanted to make a desktop operating system be more datacenter friendly. I would care so much about those decisions if they didn't constantly screw me. This stops now.

I don't want sound drivers and X11 or gnome in my datacenter operating system. I don't want the kernel developers considering how priorities and implementation decisions effect software that has no place near my database server or web server or load balancer. I don't want my scheduler to be optimized for browsing the web while watching a video. My server is to serve the Internet.

We've been working on something to scratch our itch at OmniTI. If you are pissed off like me, maybe you'll like it... coming soon.

Surge is gonna kick ass.

nospam@example.com (Theo Schlossnagle) — Tue, 09 Aug 2011 23:48:22 -0400

I haven't blogged for a while because:

I have been travelling insanely. About 80k miles this year so far.
Hacking on Circonus and (subsequently) Reconnoiter.
Providing strategic and tactical guidance on some mind blowing projects for the truly awesome clientele we have at OmniTI.
Speaking at quite a number of conferences.
Attempting to participate more in some of the open source projects I can help.
... and planning for Surge.

The little timeout I am taking to write this note isn't sufficient to dive into the truly interesting things my readers usually expect. However, I'd be a jackass if I let you go any longer without knowing that Surge is going to be friggin' awesome this year. At the end of this year, if you look back and see that you missed attending Surge you'll know what regret truly is. Just sayin'.

Everyone is to blame for this continued expectation that such magic is possible.

nospam@example.com (Theo Schlossnagle) — Sun, 24 Apr 2011 20:29:59 -0400

My opinion is that the only reason the big enterprise storage vendors have gotten away with network block storage for the last decade is that they can afford to over-engineer the hell out of them and have the luxury of running enterprise workloads, which is a code phrase for “consolidated idle workloads.” When the going gets tough in enterprise storage systems, you do capacity planning and make sure your hot apps are on dedicated spindles, controllers, and network ports.

It was fantasy believing it was possible to pull off a centralized network block storage service in a multi-tenant cloud without any of the architecture shenanigans our enterprise brethren do and think that applications, databases, and business could depend on its being perfect. Honestly, we should have know better. We the applications developers asked what is perhaps the crappiest of all abstractions in computers to solve all of our availability problems for us. We asked for magic. Clearly, the vendor never should have made the promise of magic, but everyone is to blame for this continued expectation that such magic is possible.

- Mark Mayo of Joyent

This is from an interesting article by Mark Mayo over at Joyent with some excellent gems in it.

Surge 2011 CFP Extension

nospam@example.com (Theo Schlossnagle) — Tue, 05 Apr 2011 22:41:11 -0400

The CFPs have been rolling in for Surge 2011; these are exciting times. It does, however, appear that our description of what we're looking for has produced a different set of submissions that what I expected. I think it might help to better understand what sessions were like last year and, luckily, we'll be releasing all of the Surge 2010 video footage this week. I apologize for the poor audio quality, we intend to pull in A/V recording professionals this year.

We've received some great proposals! However, a surprising amount of them are presentations about products. This conference is about problems and their solutions. It is a conference for practitioners by practitioners. Blood... in the mud. I want to (as we did last year) share our struggles for better collective experience.

Emphasis: Accepted proposals will demonstrate real-life scalability challenges and creative solutions. We love case studies and learning from our mistakes.

As such, with this context, I'm extending the Surge 2011 CFP deadline to April 17th.

Top 5 "software to saas" culture shocks.

nospam@example.com (Theo Schlossnagle) — Tue, 01 Mar 2011 21:54:54 -0500

At OmniTI, I've been a part of writing a lot of open source software, my fair share of closed source software. Some of it has been shipped and some of it has been operated as a service. While it is possible (and quite useful) to take what one learns in one scenario and apply it to another, some things simply translate poorly.

I do a lot of consulting with traditional software companies that are looking to make a transition to the new world of SaaS. Again and again, I see the same "shockers" hit these companies during their metamorphosis. Read my top 5 things you need to know if you are going to change your business model from traditional software engineering to SaaS delivery.

Using Esper to manage real-time data.

nospam@example.com (Theo Schlossnagle) — Fri, 04 Feb 2011 14:01:10 -0500

I'm flying back from a wonderful event: Strata.

I gave a talk there called "Esperwhispering" that seemed to pique many people's interest. This is the stuff you do when a database just doesn't have the horsepower to answer your questions fast enough.

Esper is an excellent, open-source CEP tool. It's a shame its GPL, but hey... you can't win 'em all.

We use esper to power many things internally at OmniTI and our clients and Esper is the code CEP engine we use to make sure Circonus custsomers know when "things go wrong."

This presentation gives some insight into what it does and why one would use it. The live presentation, of course, had more information and live demos, but... you had to show up for that (or watch the live stream of the event).

Esperwhispering

View more presentations from postwait.

#ywahusty

nospam@example.com (Theo Schlossnagle) — Fri, 04 Feb 2011 10:41:44 -0500

Defining the term:

I recently used a term and was hit with a lot of out-of-band requests for explanation. It's a good one and excellent food for thought.

ywahusty (yuh-wuh-hus-tee): you will always have users smarter than you.

This basic concept is one of sound, pragmatic systems engineering that might appear to fly in the face of traditional product engineering... but doesn't.

In traditional product engineering, there is a goal to produce a product that is both accessible and useful to the largest subset of the predefined audience of the product. Basically, in software terms: you don't put too many knobs on the dashboard or you will have all sorts of operator errors. You work very hard to tune those knobs or have then auto-tune so that the product behaves correctly under normal operational uses (and of course, their edge conditions). This is just good product design.

We are faced with a problem: shit goes wrong. As an engineer for a software product, you know full well that shit will break. Those knobs will be set and/or auto-tuned and you'll be thinking: "what if?" The: "Wow, I've never seen this set of inputs. The min_wu_tang_threshold is 11... it might behave better at 42 right now... sure wish I could twist that knob that product management insisted I leave off right now." The truth is: the product you ship is not the product they receive. Not every knob you ship has to be in the "manual," it can be in the unsupported "ninja manual." You can absolutely instrument your application without sacrificing simple and intuitive use by the majority of your audience.

Bottom line? The knob should be there. My hat goes off to an engineering team that takes this to heart: "I am ten ninjas." And a second tip of the hat to the diaspora of Solaris kernel engineers that have allowed us to tweak (seemingly) insane operational constraints on production systems without applying patches or rebooting.

Appealing to the pragmatist.

I'll attempt to appeal to the pragmatist in you. You are equipped with more knowledge now than you were when you started reading this prose. This is a truth whether you agree with anything I've said simply because you have more information. Even you, with you current implementation knowledge would benefit from the ability to tweak, twist and manipulate the parameters of your operating product because it will allow you to test new ideas, verify assumptions and validate the design and implementation choices you've made with far greater ease.

As a side note, gdb and/or mdb aren't the "knobs" I'm talking about: think more along the lines of sysctl's on most UNIX implementations. These are (mostly) run-time tunable parameters -- and yes users can set them to values that will cause certain, and often immediate, combustion. (good) Kernel engineers know that price benefit of having these knobs far outweighs their liability.

Appealing to the experienced.

Now, we come full circle, for the honest engineers that can check their ego at the door. I have never in my life developed a piece of successful software that had a user-base that contained only users dumber than myself. The concept therein is absurd for any project with even moderate adoption. You engineer your systems to be both observable and operable because at the end of the day you will get schooled by (at least some of) your users.

Again... please make your software operable. Why? #ywahusty

Why haven't all my graphs been useful like this?

nospam@example.com (Theo Schlossnagle) — Thu, 20 Jan 2011 16:21:28 -0500

I don't know now many times I've tried to eyeball a graph from this week against a graph from last week. This is both painful and senseless as my eyeballs simply aren't that good. Enter "the right way to do it." Bliss.

Capacity Planning

nospam@example.com (Theo Schlossnagle) — Wed, 05 Jan 2011 19:33:46 -0500

I used to export data into Excel and do linear regressions on it... How I have no need. I am happier person and people like me more.

Here's how we made capacity planning easier for everyone.

Theo seeks aspiring database administrator

nospam@example.com (Theo Schlossnagle) — Tue, 28 Dec 2010 08:20:40 -0500

What does it mean to be a database administrator? It means more than just respecting data in a room full of engineers and analysts that do not -- it means bending them to your will and making them respect it, too. It means knowing when people's concept of sacrosanct data integrity and consistency can be thrown out the window and letting those people know that, this time, they can cope with inaccuracy or volatility. You think those things contradict? This job's not for you.

Do you think the world can be solved through relational algebra? I'm listening. Do you think the world should be solved through relational algebra? Don't apply here. Have you edited a database redo/write-ahead log in a hex editor to bootstrap a failed recovery? Now we're talking about reality. We live in a world where the probability of bit errors on disks rattles our cage by guaranteeing corrupt data and indexes when using traditional filesystems. Being responsible for data in this world is no simple task.

If you think you can scale single RDBMS to the sizes we see on a daily basis, I'll beat you with a stick.
If you think RDBMS is are passe and any problem can be appropriately solved with a NoSQL solution, I'll beat you with a stick.
If you think data is holy, I'll buy you a stick.
If you try to pipe my data through the wrong tool, I'll carve zeros and ones into my stick and then beat you with it.
If you think its okay to let someone who cares more about code than data roll out queries without your review, I'll make you beat them with a stick, then beat you with a stick while you clean up their bloody remains.

In a real database architect's role, you don't manage a database -- you manage data. You keep it secret, you keep it safe, you keep it clean, and you keep it available. From Oracle, to PostgreSQL, to MSSQL, to MySQL, to Cassandra, to Riak, to Hadoop, to Esper you will be responsible for the operational integrity of systems that touch data and turn it into useful outputs for people and businesses.

If you know relational databases and at least one of NoSQL systems or streaming databases (CEP/IEP), then you may have what it takes to be on the OmniTI data management team.

Web Performance Boot Camp

nospam@example.com (Theo Schlossnagle) — Tue, 02 Nov 2010 16:33:09 -0400

Slides are here, get 'em while they're hot.

Perform or Die

nospam@example.com (Theo Schlossnagle) — Fri, 29 Oct 2010 22:39:50 -0400

There's no excuse these days for crappy web performance. The low hanging fruit is just too low. It is trivial to make a web site that loads fast for users. Do it. If you don't know how, you should come to my Web Performance Boot Camp tutorial at ApacheCon. Make you site fast — no excuses.

Why JSON sucks.

nospam@example.com (Theo Schlossnagle) — Sun, 10 Oct 2010 09:19:17 -0400

JSON sucks. Don't get me wrong, I love the simplicity of it. It's simple, it's easy, it's portable, it's ubiquitous at this point. None of that means it doesn't suck. Outside of Javascript (hence the portability), JSON itself is limited to native types in the grammar:

null
object (like a hash)
array
string
integer (signed 32bit)
number (double)
boolean

Really? It's 2010 and we're all flocking to a grammar where we can only accurately represent integers up to 2³¹-1. WTF? Any new standard we adopted should certainly have had grammar specifications for: 1, 8, 16, 32, 64, 128 bit signed and unsigned integers, 32bit and 64bit IEEE floating point (float and double) as well as arbitrary precision real numbers. I'd much rather have a grammar that I struggle to translate into my native language data types (like, "hmmm, what am I supposed to do with a real in C?") than I would have a grammar in which I cannot precisely express myself.

Every time I need to (correctly) represent a large integer such as 4611686018427387900, I'm forced to do so in a string. It causes me to throw up in mouth a little. Everyone seems dead set on this, so I suppose I'll learn to cherish the flavor of bile.

Finding leaks on Solaris (w/o Valgrind)

nospam@example.com (Theo Schlossnagle) — Sat, 09 Oct 2010 16:04:18 -0400

Premise: I write a lot of C code. I run a lot of Solaris.

Sadness: One of my favorite tools ever made is Valgrind. Valgrind does not run on Solaris.

A lot of the C code I write is event-driven and as such (complicated) it is harder to write code and leaked memory is a common residual of this more complicated coding effort. Memory leaks suck. Most of the code I write is systems-level code and as such is exercised heavily and needs to run for months or years without restart. So, leaks more than suck.

Valgrind will tell you exactly where you're leaking (and it feels like it tells you why)... BFM.

On Solaris, we have a different option: libumem.

I compile almost all of my apps against libumem, which has the effect of replacing malloc/free/and friends with a more multi-processor scalable slab-allocator implementation than the default one in libc. A very useful feature in libumem is debugging and leak detection... how?

First, let's assume you've compiled your app with debugging symbols (-g in the compiler flags). Next, I'll assume you didn't have the foresight to link against libumem (-lumem on the linker line). We need to link in libumem and we need to turn on debugging. In the case of this example, our app is the Apache web server httpd.

LD_PRELOAD=libumem.so.1 UMEM_DEBUG=default /opt/apache22/bin/httpd

Now, just find the process ID of exampled (perhaps as easy as a pgrep exampled). It is leaking, so we'll use the process ID 666. To find the leaks (without killing or restarting), we can simply use the mdb umem helper ::findleaks on the running process.

echo "::findleaks -d " | mdb -p 666 > leak-profile.txt

Note you can also do this from a core... very nice.

The output has a summary and then individual profile records grouped by stack trace of allocation, like:

umem_alloc_16384 leak: 203 buffers, 16384 bytes each, 3325952 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
         1d36b60          1d37000   70235c07811a50              197
                           65a468           617900                0
                 libumem.so.1`umem_cache_alloc_debug+0x152
                 libumem.so.1`umem_cache_alloc+0x1a2
                 libumem.so.1`umem_alloc+0xdb
                 libumem.so.1`malloc+0x59
                 libapr.so.0.2.12`allocator_alloc+0x466
                 libapr.so.0.2.12`apr_palloc+0xda
                 bio_filter_out_ctx_new+0x37
                 ssl_io_filter_init+0xb1
                 ssl_init_ssl_connection+0x2cd
                 ssl_hook_pre_connection+0x123
                 ap_run_pre_connection+0xc3
                 ap_process_connection+0x30
                 process_socket+0xa8
                 worker_thread+0x2d8
                 libapr.so.0.2.12`dummy_worker+0x30

mod_ssl... Why are you so mean to me?