My typical blog job posting is all about sticks. OmniTI is a professional services organization; in short, we’re consultants. However, where most consultants practice “consult and run,” we at OmniTI are about building long-term relationships. The majority of our consulting gigs convert from a few month engagement into multi-year contracts because of our intense commitment to customer satisfaction. Instead of threatening to beat you with a stick just yet, I’m going to tell you some neat things we’ve done over the last couple of months.

The Position

I’m looking for an SA. Now, SA normally stands for Systems Adminstrator and some SAs think their roles are clearly defined. Here we clearly define responsibilities, not roles. An SA here is an expert in “whatever it takes” to make our clients’ productions systems “just work.” So, what is it that we do? This is best illustrated by what it is that we on the ops team have done over the last several months.

  • Planned and executed an internal office move including: spec’ing the new data room power and A/C, coordinating PRI and phone line moves, Asterisk reconfigurations, router installs, and the usual bit of bare metal recovery.
  • Installed a 1+TB PostgreSQL instance, a 6TB (planned capacity) PostgreSQL instance, disaster recovery management and rollout of a 1TB Oracle system on enterprise hardware.
  • Performed a few whacky and drastic LiveUpgrades on Solaris.
  • Managed the migration of several large scale systems from Linux to Solaris.
  • Managed the migration of several large scale systems from Solaris to Linux
  • Implemented hundreds of custom business monitors and integrated them with our monitoring systems.
  • Built the zetaback ZFS backup and recovery tool.
  • Authored custom modules for Asterisk.
  • Managed 40 independent Subversion respositories.
  • backported bits of OpenSolaris kernel modules to Solaris 10.
  • built countless custom RPM packages and Solaris SVR4 packages for production deployments.
  • Installed a VMWare based training lab.
  • Asked questions only answerable by DTrace
  • Spent some time on a ladder in a ceiling looking for cat6e.
  • Articulated vastly complex email policy using the Sieve++ language in Ecelerity for workloads in excess of 20 million messages per day.
  • In a given day, I’ll touch:
    • OpenSolaris, Solaris {6,7,8,9,10}, at least 5 flavors of Linux, FreeBSD {4,5,6}, OpenBSD {3,4}, IOS {11,12}, Foundry switch OS 7, Mac OS X, OpenVZ, VMWare, Solaris Zones;
    • md, LVM, VxVM, SVM, NFS, ext2, ext3, XFS, VxFS, ufs, and ZFS;
    • ssh, OpenVPN, SMF, PostgreSQL 8, Oracle {8,9}, MySQL {3,4,5}, SQLite 3, PowerDNS, Bind {8,9}, Postfix, Exim, Sendmail, Ecelerity, Cyrus, Dovecot, Sieve, Spread, Wackamole, Apache {1,2} (and just about every module), Squid {2,3}, Lighttpd, Subversion, Trac, Asterisk, Amanda, Veritas NetBackup, PHP {4,5}, Perl 5, C.
    • Coffee… a lot of coffee.
  • Some drywall work.

Being consultants gives us unique challenges and opportunities. We get to see a world of problems unlike any non-consulting business. We get a super-aggrgegate of all the problems from all our clients. As I like problems, this is hands-down the coolest part of the job. The opportunity to work with a mind-boggling variety of technologies is also very very cool.

So, no you are wondering: where’s the part about beating me with a stick? Well, Eric manages the operations team here. He’s a crazy viking dude who it 6’4” and hits a lot harder than me. My stick is the last thing you need to worry about.

We have a great team, hard problems and high expectations. We take our clients’ problems very seriously and take pride in the solutions we build to solve them.

As with all operations jobs, this isn’t for the shy. You wear a pager; you go on-call; you arrive first after catastrophic failures; you get to be a hero.

To apply, send your resume and a compelling story to

Obligatory stick-beating prose

From my last posting, I’ll excerpt the following:

  • If you call having spare parts having a “failover,” I’ll beat you with a stick.
  • If you have important data that doesn’t sport a backup and retention policy, I’ll make you write oxymoron on the white board a hundred times, then I’ll beat you with a stick.
  • If you call a service production and you don’t get paged when it breaks, I’ll page you – with a stick.
  • If you think documenting how you install and configure software jeopardizes job security, I’ll beat you with two sticks: the mighty stick of literacy and the stinging cane of vanity.
  • If the idea of your operating systems kernel panicking when you try to mount your database’s filesystem makes you want to wet your pants, bring a towel to work.

And I’ll add to this:

  • If you think production downtime is “okay,” Eric will beat you with a stick, I’ll beat you with a stick, the operations team will take turns beating you with sticks and we’ll invite the client over for our piñata party.

To quote a great Red Stripe:

If you want to argue we’ll have a stick-fight…

It will be an old-fashioned Jamaican stick-fight…

and I will win.