Esoteric Curio - PostgreSQL

The myopic focus on IT and engineering has to stop.

nospam@example.com (Theo Schlossnagle) — Mon, 27 Aug 2012 07:30:49 -0400

Business is king. Customers rule. Service is everything. Yet every organization I go into has an engineering group that can't see outside their bubble. Perhaps they can, but they certainly choose not to.

I'm an engineer, I write code. I've written approaching 100k lines of C code in my life time, I've administered tens of thousands of systems in my career and I've help plan some of the largest customer-facing infrastructure ever built. I've learned a tremendous amount about technology and the hubristic nature of engineering teams. The most important take away from all of this? The technology doesn't mean anything unless it enables business by providing better service to customers.

Now, I realize that when I rant about this to technology folk, they emphatically agree. But, I'm tired of the lip service. People today in architecture, engineering and operations say again and again that their focus on enabling better customer experience. It's a nice sentiment, but every time I dive into someone's instrumentation and monitoring, I see an absolute vacuum when it comes to non-IT data.

The obvious things like financial and customer service metrics are missing, but so are all the more subtle things. Hiring is hard; finding and retaining talent is challenging; providing good benefits that add value and increase job appeal is a competitive task. All of these things are critically important to the organization as a whole (and specifically engineering and IT) and yet they are completely absent from the "monitoring" within the organization.

The truth is that there is absolutely critical telemetry coming from every facet of your organization. All of this telemetry is either directly related to providing better service to customers or directly related to providing better service to your organization itself which, in turn, stabilizes the platform on which you deliver products and services. Of this, I shouldn't have to convince you and I find that no convincing of the general population is required. Yet, here we are with almost every organization I see standing blind to this vital information.

Don't get me wrong, I don't think technology isn't a first-class component of today's (and tomorrow's) organizations. In fact, I think the technology group has been applying radically advanced techniques to telemetry data for years. It's high time that these techniques and tools were applied to the organization unabridged.

There is a profound shift in data transparency and accountability coming to the organization to tomorrow. If you don't buy in, you'll simply fail to achieve the agility and efficiencies of your competition. I'm here, with Circonus, to make that happen.

Business is king, not engineering. The difficult (but exceptionally simple) shift of engineering's focus from serving itself to serving the business as a whole will remake IT as the engine of the organization. As soon as you embrace this shift, technology will be the most powerful tool your organization has at its disposal.

Creativity and Execution

nospam@example.com (Theo Schlossnagle) — Tue, 10 Jul 2012 13:33:41 -0400

I've built a few successful products and looking back on their success, I think that the mantra that drove product development is what separated our products from the rest of the market: "products built from pain." All of the products we've built were done so to relieve acute pain. Not pain we researched; pain we experienced. We built products and changed the world of software because our lives sucked.

Photograph by AlexRK.

I've read a lot of books lately on new ways of running organizations and different methods of motivating people and many of them focus on studies around jobs that require a tremendous amount of creativity or "thought workers." I think these books are interesting, exciting and I'm interested in carefully applying some of the research at my places of work. Thinking is critical, creativity is what enables you to innovate. Now, I'll say something highly unpopular: get over yourself.

Inspiration to drive innovation is what we're looking for and most innovation happens through punctuated equilibrium. Inspiration, by its very nature, is tied to a punctuated result because it is sudden. We often speak of "flashes" of inspiration and one definition of the word is even "a sudden, brilliant or timely idea."

If you've ever had a stroke of genius, there is a decent chance it happened in the shower. The place where you relax, zone out, and cannot escape ideas for implementations. What I see now I find sad: people restructuring their lives to have lots and lots of what I call "shower time."

If you write code, which many of my readers do, I'm about to piss a vast majority of you off. We're not thought workers. Innovation in code is a rare thing and coding is a tedious task. Until the day when we can merely think of inputs, outputs and algorithms and the computer will simply codify them on our behalf, we all will spend a lot of time meticulously telling a computer what to do. It might be challenging. It might require focus. It requires a perfect vocabulary, impeccable grammar and a mastery of common idioms and colloquialisms, but at the end of the day 99% of it is writing a set of instructions for a deterministic system to follow. Bottom line, it is largely not a creative process. I know that many of us (myself included) like to think of new and clever ways to do something -- it's good mental exercise. But, almost every time someone shows me a new and exciting approach to solving a problem, I can find an almost identical reference implementation of the same solution published either academically or commercially.

If you step back, the large applications and services we all are building are new and different. It took vision, whimsy, courage and maybe a fair share of batshit-crazy inflated self-confidence to pursue them in the first place. I applaud this, but find it paramount that one stay grounded in the fact that executing on that vision is 99% blood in the mud. It's hard, it's often not creative, it rarely resembles "thought work."

I see more and more people trying to avoid the sweat and focus on constructing halcyon environments where they have the freedom to think differently and be creative. Here's the deal, you're a human being and as such, you don't need a special environment to arrive at freedom of thought; nothing and no one can take that away from you but you. It is the constraints that result in true creativity. It's wading through shit for 18 hours, suffering your own bad decisions and those of others, arriving at a moment of painful failure and tears that ultimately requires a shower. It's not the shower you took yesterday or the day before. It won't be the shower you take tomorrow. This shower is truly "shower time." In this shower, maybe (just maybe), you've set the stage for sudden, brilliant or timely idea - because you've earned it.

Peaches and Pecans

nospam@example.com (Theo Schlossnagle) — Wed, 13 Jun 2012 12:21:19 -0400

Peaches and pecans on vanilla ice cream is a wonderful thing, but get some perspective on how you came to enjoy it. I have heard (and have told others), “life is too short to do something you don’t enjoy,” but the truth is there is no way to revel in everything you do at every moment; not even the most ambitious and determined hedonist can achieve this. While I don’t think he was right about everything, I feel confident Sigmund Freud nailed this one: “We are so made, that we can only derive intense enjoyment from a contrast and only very little from a state of things.” So, not only can we not enjoy everything, that which we do enjoy is aided by the lack thereof in other things. Pitting the peaches, cracking the nuts, even making the ice cream heightens the experience of its ultimate, decadent demise.

How does this relate to work? In more ways than you’d think. Most things we do have small parts that we don’t enjoy; it is part of being complete. I’ll give a few examples of regular things I do that are nowhere near the pinnacle of my excitement and satisfaction curves.

I am an engineer and I love to solve problems. The more difficult, the more rewarding. Increasing the difficulty increases the likelihood of failure, in turn making success more exultant. I write code; I’m good at it. Part of writing code includes considering aspects of who owns the code, how it is licensed, and who protects the user from claims of intellectual property infringement. That’s right, I said “part of writing the code.” If I am skipping these tasks, I am increasing the risk for every consumer of my code. It’s part of my job, and (for me) is certainly not the most enjoyable part.

Not everyone has these responsibilities, but I’d bet if you think about your job for a few minutes you can name a handful of responsibilities that, for lack of a more eloquent description, suck. That’s just one example. I also have to manage people, review financials, work with banks and lawyers, hire, fire, interact with clients, market and sell... oh yeah, and solve technical problems. Which of these do I enjoy the least? That’s my secret and if I do my job well, you’ll never know. I try to embrace the parts of my job that do not resonate with my strengths and leverage them to become stronger.

Parts of your job are going to suck, that’s just how it is. Now, you might be able to delegate some of these various tasks, but I have found that embracing some of these things and making them truly a part of what you deliver increases the quality of your work, makes your successes more satisfying and generally makes you better at what you do. We tend to not enjoy tasks that aren’t in our wheelhouse. Things that make us uncomfortable are the things that make us grow.

Here’s the rub: you can grow up or grow out. Growing “up” is challenging yourself to harder and harder tasks in your hyperfocused discipline. Growing “out” is pursuing and accepting challenges that are related to your business, but not “your job.” Upward growth makes you better, more expert, and more elite. Outward growth makes you better, more instrumental in overall success and an invaluable player in the business as a whole. Upward growth is far more comfortable and less intimidating; it’s the known unknown and failure is less likely. To be the best you need to invest in both.

courtesy of alaczek

Life truly is short and not enjoying what you do (at all) is a vast waste of life itself and so it is a balancing act. How much of your day-to-day job should be the stuff that you don’t like (outward growth that makes you uncomfortable)? The idealists out there are simply going to hate this answer: non-zero. I’ll get more specific for those that are still reading. “Let me do what I do. I’m good at it and that’s what you hired me for.” Sound familiar? I say bullshit. I didn’t hire you to do X. I might have hired you because you demonstrated that you were competent at doing X, but I hired you to make my team a better team, my product a better product, and my business a better business.

I’ve spent a good deal of my time searching for balance and here’s what I found. Everyday that I spend time doing the things that I love and the things at which I feel most capable I feel awesome; I feel successful and satisfied... in the short run. A week goes by and I reflect on all that I’ve accomplished and I see that the business would profited more had I focused on those things that made me uncomfortable, required focus and personal growth (that which I was less confident about executing flawlessly).

As a contrapositive, when I don’t focus on upward growth and instead focus on all the various places I can add value outside my core expertise, I see the future brighten by the hour. I add value. I add real, tangible value to initiatives through my learning and unique perspective. Business is better, teams are stronger, and clients are happier. The goals are great, but I can’t always walk a path whose journey does not yield deep personal satisfaction; I become uninspired, my passion drops, I lose my perspective and become ineffective.

It’s a balancing act indeed and for me that balance is 80/20. I find, looking back at the last several years, that when I fall below 20% of my time in outward growth I lose potential value and perspective. If I fall below 80% upward growth, I lose passion and perspective. Notice that? The thing I always lose when I lose balance is perspective. This 20% may not be the right number for you, so you should strive to discover your own balance -- just don’t lie to yourself, because you’ll lose in the long run. I said I learned this through failure. I never seem to achieve a consistent 20/80 split and my moments of equilibrium are fleeting, but I feel them as they slide by.

I’m not saying that pitting peaches (which I truly hate) or shelling pecans (which I find devoid of mental stimulation) or making ice cream (which is good fun) is necessary for the enjoyment of a most wonderful peaches and pecan sundae. I’m saying that if I do it 1 out of 5 times, I am more discerning, more moderate in my consumption and derive far more satisfaction from every dessert I eat.

Peaches and pecans on vanilla ice cream is a wonderful thing, but get some perspective on how you came to enjoy it.

OmniOS

nospam@example.com (Theo Schlossnagle) — Thu, 12 Apr 2012 15:04:01 -0400

And this is it ... OmniOS.

The desktop and server: oil and water.

nospam@example.com (Theo Schlossnagle) — Fri, 23 Mar 2012 21:48:20 -0400

I've been using these computer things for a while. I've written what is now over 100k lines of production C code and many thousands of lines of code in a variety of other languages. I've seen my software run and I've run other people software. One thing they all have in common is their propensity to break under unforeseen circumstances. Shit happens.

On my laptop, I don't care much. I want nice, I want convenient, I want new and pretty and productive. I'm willing to tolerate a nominal amount of instability for those desires. I have to reboot my Macbook Air at least once every month to accommodate general software failures and security updates. There is, after all, a lot of crazy software on my workstation. (yes I call my laptop a work station as it is the station at which I work -- how many of your do the same?)

Somehow, somewhere some engineer and their cult following decided that they wanted to make a datacenter operating systems be more desktop friendly and a product manager somewhere and their hefty company decided they wanted to make a desktop operating system be more datacenter friendly. I would care so much about those decisions if they didn't constantly screw me. This stops now.

I don't want sound drivers and X11 or gnome in my datacenter operating system. I don't want the kernel developers considering how priorities and implementation decisions effect software that has no place near my database server or web server or load balancer. I don't want my scheduler to be optimized for browsing the web while watching a video. My server is to serve the Internet.

We've been working on something to scratch our itch at OmniTI. If you are pissed off like me, maybe you'll like it... coming soon.

Surge is gonna kick ass.

nospam@example.com (Theo Schlossnagle) — Tue, 09 Aug 2011 23:48:22 -0400

I haven't blogged for a while because:

I have been travelling insanely. About 80k miles this year so far.
Hacking on Circonus and (subsequently) Reconnoiter.
Providing strategic and tactical guidance on some mind blowing projects for the truly awesome clientele we have at OmniTI.
Speaking at quite a number of conferences.
Attempting to participate more in some of the open source projects I can help.
... and planning for Surge.

The little timeout I am taking to write this note isn't sufficient to dive into the truly interesting things my readers usually expect. However, I'd be a jackass if I let you go any longer without knowing that Surge is going to be friggin' awesome this year. At the end of this year, if you look back and see that you missed attending Surge you'll know what regret truly is. Just sayin'.

Everyone is to blame for this continued expectation that such magic is possible.

nospam@example.com (Theo Schlossnagle) — Sun, 24 Apr 2011 20:29:59 -0400

My opinion is that the only reason the big enterprise storage vendors have gotten away with network block storage for the last decade is that they can afford to over-engineer the hell out of them and have the luxury of running enterprise workloads, which is a code phrase for “consolidated idle workloads.” When the going gets tough in enterprise storage systems, you do capacity planning and make sure your hot apps are on dedicated spindles, controllers, and network ports.

It was fantasy believing it was possible to pull off a centralized network block storage service in a multi-tenant cloud without any of the architecture shenanigans our enterprise brethren do and think that applications, databases, and business could depend on its being perfect. Honestly, we should have know better. We the applications developers asked what is perhaps the crappiest of all abstractions in computers to solve all of our availability problems for us. We asked for magic. Clearly, the vendor never should have made the promise of magic, but everyone is to blame for this continued expectation that such magic is possible.

- Mark Mayo of Joyent

This is from an interesting article by Mark Mayo over at Joyent with some excellent gems in it.

Surge 2011 CFP Extension

nospam@example.com (Theo Schlossnagle) — Tue, 05 Apr 2011 22:41:11 -0400

The CFPs have been rolling in for Surge 2011; these are exciting times. It does, however, appear that our description of what we're looking for has produced a different set of submissions that what I expected. I think it might help to better understand what sessions were like last year and, luckily, we'll be releasing all of the Surge 2010 video footage this week. I apologize for the poor audio quality, we intend to pull in A/V recording professionals this year.

We've received some great proposals! However, a surprising amount of them are presentations about products. This conference is about problems and their solutions. It is a conference for practitioners by practitioners. Blood... in the mud. I want to (as we did last year) share our struggles for better collective experience.

Emphasis: Accepted proposals will demonstrate real-life scalability challenges and creative solutions. We love case studies and learning from our mistakes.

As such, with this context, I'm extending the Surge 2011 CFP deadline to April 17th.

Top 5 "software to saas" culture shocks.

nospam@example.com (Theo Schlossnagle) — Tue, 01 Mar 2011 21:54:54 -0500

At OmniTI, I've been a part of writing a lot of open source software, my fair share of closed source software. Some of it has been shipped and some of it has been operated as a service. While it is possible (and quite useful) to take what one learns in one scenario and apply it to another, some things simply translate poorly.

I do a lot of consulting with traditional software companies that are looking to make a transition to the new world of SaaS. Again and again, I see the same "shockers" hit these companies during their metamorphosis. Read my top 5 things you need to know if you are going to change your business model from traditional software engineering to SaaS delivery.

Using Esper to manage real-time data.

nospam@example.com (Theo Schlossnagle) — Fri, 04 Feb 2011 14:01:10 -0500

I'm flying back from a wonderful event: Strata.

I gave a talk there called "Esperwhispering" that seemed to pique many people's interest. This is the stuff you do when a database just doesn't have the horsepower to answer your questions fast enough.

Esper is an excellent, open-source CEP tool. It's a shame its GPL, but hey... you can't win 'em all.

We use esper to power many things internally at OmniTI and our clients and Esper is the code CEP engine we use to make sure Circonus custsomers know when "things go wrong."

This presentation gives some insight into what it does and why one would use it. The live presentation, of course, had more information and live demos, but... you had to show up for that (or watch the live stream of the event).

Esperwhispering

View more presentations from postwait.

#ywahusty

nospam@example.com (Theo Schlossnagle) — Fri, 04 Feb 2011 10:41:44 -0500

Defining the term:

I recently used a term and was hit with a lot of out-of-band requests for explanation. It's a good one and excellent food for thought.

ywahusty (yuh-wuh-hus-tee): you will always have users smarter than you.

This basic concept is one of sound, pragmatic systems engineering that might appear to fly in the face of traditional product engineering... but doesn't.

In traditional product engineering, there is a goal to produce a product that is both accessible and useful to the largest subset of the predefined audience of the product. Basically, in software terms: you don't put too many knobs on the dashboard or you will have all sorts of operator errors. You work very hard to tune those knobs or have then auto-tune so that the product behaves correctly under normal operational uses (and of course, their edge conditions). This is just good product design.

We are faced with a problem: shit goes wrong. As an engineer for a software product, you know full well that shit will break. Those knobs will be set and/or auto-tuned and you'll be thinking: "what if?" The: "Wow, I've never seen this set of inputs. The min_wu_tang_threshold is 11... it might behave better at 42 right now... sure wish I could twist that knob that product management insisted I leave off right now." The truth is: the product you ship is not the product they receive. Not every knob you ship has to be in the "manual," it can be in the unsupported "ninja manual." You can absolutely instrument your application without sacrificing simple and intuitive use by the majority of your audience.

Bottom line? The knob should be there. My hat goes off to an engineering team that takes this to heart: "I am ten ninjas." And a second tip of the hat to the diaspora of Solaris kernel engineers that have allowed us to tweak (seemingly) insane operational constraints on production systems without applying patches or rebooting.

Appealing to the pragmatist.

I'll attempt to appeal to the pragmatist in you. You are equipped with more knowledge now than you were when you started reading this prose. This is a truth whether you agree with anything I've said simply because you have more information. Even you, with you current implementation knowledge would benefit from the ability to tweak, twist and manipulate the parameters of your operating product because it will allow you to test new ideas, verify assumptions and validate the design and implementation choices you've made with far greater ease.

As a side note, gdb and/or mdb aren't the "knobs" I'm talking about: think more along the lines of sysctl's on most UNIX implementations. These are (mostly) run-time tunable parameters -- and yes users can set them to values that will cause certain, and often immediate, combustion. (good) Kernel engineers know that price benefit of having these knobs far outweighs their liability.

Appealing to the experienced.

Now, we come full circle, for the honest engineers that can check their ego at the door. I have never in my life developed a piece of successful software that had a user-base that contained only users dumber than myself. The concept therein is absurd for any project with even moderate adoption. You engineer your systems to be both observable and operable because at the end of the day you will get schooled by (at least some of) your users.

Again... please make your software operable. Why? #ywahusty

Why haven't all my graphs been useful like this?

nospam@example.com (Theo Schlossnagle) — Thu, 20 Jan 2011 16:21:28 -0500

I don't know now many times I've tried to eyeball a graph from this week against a graph from last week. This is both painful and senseless as my eyeballs simply aren't that good. Enter "the right way to do it." Bliss.

Capacity Planning

nospam@example.com (Theo Schlossnagle) — Wed, 05 Jan 2011 19:33:46 -0500

I used to export data into Excel and do linear regressions on it... How I have no need. I am happier person and people like me more.

Here's how we made capacity planning easier for everyone.

Theo seeks aspiring database administrator

nospam@example.com (Theo Schlossnagle) — Tue, 28 Dec 2010 08:20:40 -0500

What does it mean to be a database administrator? It means more than just respecting data in a room full of engineers and analysts that do not -- it means bending them to your will and making them respect it, too. It means knowing when people's concept of sacrosanct data integrity and consistency can be thrown out the window and letting those people know that, this time, they can cope with inaccuracy or volatility. You think those things contradict? This job's not for you.

Do you think the world can be solved through relational algebra? I'm listening. Do you think the world should be solved through relational algebra? Don't apply here. Have you edited a database redo/write-ahead log in a hex editor to bootstrap a failed recovery? Now we're talking about reality. We live in a world where the probability of bit errors on disks rattles our cage by guaranteeing corrupt data and indexes when using traditional filesystems. Being responsible for data in this world is no simple task.

If you think you can scale single RDBMS to the sizes we see on a daily basis, I'll beat you with a stick.
If you think RDBMS is are passe and any problem can be appropriately solved with a NoSQL solution, I'll beat you with a stick.
If you think data is holy, I'll buy you a stick.
If you try to pipe my data through the wrong tool, I'll carve zeros and ones into my stick and then beat you with it.
If you think its okay to let someone who cares more about code than data roll out queries without your review, I'll make you beat them with a stick, then beat you with a stick while you clean up their bloody remains.

In a real database architect's role, you don't manage a database -- you manage data. You keep it secret, you keep it safe, you keep it clean, and you keep it available. From Oracle, to PostgreSQL, to MSSQL, to MySQL, to Cassandra, to Riak, to Hadoop, to Esper you will be responsible for the operational integrity of systems that touch data and turn it into useful outputs for people and businesses.

If you know relational databases and at least one of NoSQL systems or streaming databases (CEP/IEP), then you may have what it takes to be on the OmniTI data management team.

Web Performance Boot Camp

nospam@example.com (Theo Schlossnagle) — Tue, 02 Nov 2010 16:33:09 -0400

Slides are here, get 'em while they're hot.