So a panic is a panic, right? Well yes in that is stops your server dead in its tracks, but the how you deal with it is the important thing. I’ve been frustrated for a long time with Linux and FreeBSD (Linux more so) regarding the lack of good post-mortem crash analysis facilities. Some background might be helpful here. One of the clusters that my company administrates has 10 FreeBSD 5.3, 1 FreeBSD 5.2.1, 1 FreeBSD 5.5, 2 Linux CentOS 4.1 and 2 Solaris 10 machines running (all dual Xeons). And we get between 1 and 3 kernel panics per week? You might say bad hardware, or bad OSes or odd gravitational flux in the area – you’d be wrong. There is nothing wrong with all 16 machines and everyone of them takes turns kernel panicing. There are a variety of vendors (all the boxes aren’t identical). So, why so unstable? I believe our app is just pushing on these boxes in new ways. We’re looking at sustained 500+ disk ops/second and 20-30 thousand open TCP sessions pretty full use of the 2GB of RAM in each box. Where am I going? Well, FreeBSD has a feature called savecore. These machines are 400 miles away from our admin team and I can’t (even with the help of several core FreeBSD developers) get savecore to function reliably. So, when one of these panics occurs, more often than not, I have nothing to work with upon reboot. Sigh. FreeBSD sucks right? No, take a look at Linux to see bad. the box panics or hangs. I can’t leave it hung to do online debugging through the kernel debugger. Besides, I might not think of the crucial information I want to see until a day later. I need a core dump and for the life of me, there is no way to get a good port-mortem system image and vmcore. WTF is going on here? How can people run this stuff in heavy production environments when there is no really sound way to troubleshoot kernel faults without attempting to repeat them. Let’s demonstrate this. If I have a kernel panic and get some sort of complicated register dump and stack trace, should I (a) cut and paste it and post it to a mailing list or (b) take a picture of my monitor with my digital camera. Well, if you answer (b) you are retarded (take offense, please). The fact that digital cameras are common linux crash dump reporting tools is truly sad. Now take a look at how many Solaris users use option (b). So… I covered Linux and FreeBSD. What about Solaris you ask? (If you didn’t ask, you should). Upon panic (which happens more rarely than on FreeBSD and Linux in our cluster) I get a reboot and all services are auto-restored before I can blink (thanks smf), but low-and-behold a present! Just for me! in /var/crash/nodename/. A complete vmcore.0 and unix.0 allowing for thorough offline (or online) post-mortem analysis. It’s like I am using a real UNIX again. Our Linux and FreeBSD installs outnumber out Solaris installs 20 to 1 at least. This rant to tell people that just because Linux is popular, doesn’t mean it can’t learn a few things from Solaris. The code is out there, grok it, adopt the methodologies and and good engineering practices. I like Linux, but I need savecore, I need it now, and I need it to unquestioningly, reliably work out-of-the-box.