Um... time to get a new HDD? Or was it just some software that gone wrong and decided to trash everything?

I'm not sure what caused the failure. ide-smart shows the drive as operating under acceptible parameters. I've gone through and reinstalled packages which had missing files. Everything should be working fine now. No user data looks to have been permenantly damaged or lost. Strangely, a lot of the files were from tetex, which is something that is never used on this system (it just came with the distro). There were also files from Perl and libwww that had damaged inodes. There did not appear to be any damaged Apache or PHP files, but I've re-downloaded and recompiled both just to be safe.

Is it actually more often a hardware failure that causes filesystem corruption? I was more under the impression that misbehaving programs are more often the culprit. Then again, my experience is more in the desktop realm (as opposed to servers).

I've actually experienced both types of disk corruption problems almost simultaneously: I once tried to recover a program from a degraded floppy disk. I then made the mistake of trying to run it, at which point it happily gobbled up my hard drive's filesystem. Fortunately, my dad used to periodically have me spend hours backing up our sizeable 40 megabyte MFM hard drive onto 5.25" floppy disks, so we were back up and running within a few hours. Must have been a dozen years ago, but I learned my lessons:
1. It pays to make backups (but I still never do).
2. Never run any program that may be corrupted or infected. Just don't do it - it's not worth the risk.
3. 40 megabytes is big when you're backing it up onto floppy disks.

As for SMART, I've never seen it report a bad drive or any other problems, so I'm a bit skeptical about its usefulness.

Heh, I remember what it was: it was in that sliver of time between when I was still confined to my 8MHz 286 (a good 6 years old by that time - amazing how rarely you had to upgrade in the late 80s to be able to play the latest games), but after I acquired a copy of X-Wing from my cousin. I was trying to get a program working that used hard disk space as EMS (because my 286 only had 1 meg of RAM) so that I could hear the digitized speech in X-Wing on my new Sound Blaster 2.0. Soon after, we "borrowed" an old 386DX-33 (that's 33MHz) with around 16MB of RAM from my dad's work, and everything changed (in other words, I was finally able to play Doom!).

EDIT: I feel old.

HunterZ wrote:

Is it actually more often a hardware failure that causes filesystem corruption? I was more under the impression that misbehaving programs are more often the culprit. Then again, my experience is more in the desktop realm (as opposed to servers).

Programs and programs - what is a 'program'?

It depends a lot on the operating system. On a *real* operating system (e.g. Unix + Windows NT/2000/XP), ordinary programs run by ordinary users can not manipulate the filesystem directly - all filesystem operations goes through the OS and from there to the file system drivers. So any corruption by software is caused by the OS and the drivers.

Unfortunately, many, many Windows systems are configured to give the daily user administrative rights => Ability to bypass the OS, ability to install drivers, and to royally f.ck up the filesystem.

I am as bad as everyone else and log on with administrative rights 🙁 My only excuse is, that I spent 3-4 months testing the setup and drivers before I started to use my PC for real. And I haven't updated the BIOS, motherboard drivers, etc. for more than 12 months now...

History:
On Saturday 11 December 2004, the primary hard drive in the server started having problems reading sectors on the hard drive. This resulted in filesystem corruption on 12 December 2004. On 12 December 2004 the server was offline from approximately 8AM GMT until 12PM GMT while workers tried to restore the server. They were successful, and a filesystem check quarantined the bad drive blocks.
At 6:31PM GMT on 12 December 2004 the server once again started having trouble reading sectors on the drive. A scan of bad blocks on the drive indicated a substantial cascading failure of the equipment. A decision was made to conduct emergency maintainence and replace the drive.
At 9AM GMT on 13 December 2004 the server was taken offline and the drive was replaced. From 10:30AM GMT until 2:30PM GMT technicians worked to restore data and accounts from the failing drive.

All accounts have been successfully transferred to the new drive. No user accounts lost any data during this process. We continue to stress that all users should do periodic backups of their sites to prevent any data loss from occurring in the future. The server will go down one last time to permanently remove the failing drive. This downtime should last no more than 5 minutes.

BTW, by "technicians", I mean my lone sorry ass waiting hours for files to copy and patches to install. Until 7:30AM. Fuck.

Snover, i have a little suggestion for you. It's a little off-topic but here it goes anyway.

I've discovered (by common use) that if you are running some sort of Windows Operating System (and maybe any other modern O.S.) it's better to get into bios and disable IDE BLOCK IDE, IDE PREFETCH MODE and the such, because those settings slightly degrade hard disk performance. This is just my opinion and should be take with a grain of salt because is hard to explain:

+ With IDE block mode enabled hard disk is slightly faster (about %10+) but it is also LOUDER and the DISK CACHE PERFORMANCE (not speed) suffers a little. I prefer to sacrifice a little speed for a quieter, more efficient cache performance. This may extend lifespam or hard-drive a little, or maybe not. Who knows.

+ IDE Prefetch Mode. Also known as READ-AHEAD. Most Operating System do it since W9X time (and maybe earlier).

Later, dude.

I'm sorry, did someone say Windows? On a server? Are you crazy?

What! you do not have the almighty windoze. You are wise, man. 😎