There’s a great article by Ted Turner in Washington Monthly about media consolidation and its very negative side effects on our society.
Last week there was a thing on Slashdot about abused hardware stories. Since it’s been a few days and I’m above posting on Slashdot, I figured I’d impart an amusing anecdote.
I was working as a second shift operator in the server room of a manufacturing company. They did the bulk of their work on a big ass HP3000. My job was mainly to run jobs on the HP3000, change reel-to-reel tapes, etc. This company had their stuff reasonably together and recognized that the year 2000 was going to cause them some problems and they had begun planning to migrate the systems over to a more modern system years in advance.
A few feet away they had a new K class HP 9000 server in a large rack. Underneath it was a lot of storage. Lots and lots. Like a hundred gigs. They hired some Indian consulting firm to do the majority of the migration work before that sort of thing was popular.
The building we worked in was fairly new. It was built maybe three or four years before I had started working there, and the server room had the nice things you’d expect from a modern server room: backup power, excellent air conditioning, no raised floor. To handle the heavy air conditioning needs in the server room (the HP3000 had a very narrow operating window, and we had an amusing procedure1 to deal with failures), the AC unit was placed above the ceiling in the server room. The engineers wanted it over an office that was connected to the server room, but the contractors insisted that it go in the server room or else there would be all kinds of functional problems.
Well, one day there was absolute panic. The compressor had failed overnight (there wasn’t a 3rd shift operator normally) and there was a huge amount of condensed water. Directly above the HP 9000. Water was pouring down from the ceiling, cascading down the front of the HP 9000, being sucked into the machine by the intake fans and finally blown out the bottom by the exhaust fans.
Finally they were able to stop the water and turn the machine off. These machines were extremely expensive at the time. I don’t remember exact figures but I think they were in the $250k to $500k ballpark. So of course HP support was called in to fix this thing. The technician took a look at it and found that while the water had voided the warranty, the only thing that needed replacing was a single board and I think maybe the power supply.
Management went and bought another HP 9000 for their Y2K migration, but they kept the old one running as a test server. It kept working through the 3 years I worked there, and I’m sure it’s still humming happily today.
Afterward the air conditioning unit was moved to where they asked it to be in the first place.
1 The procedure for dealing with a failed air conditioning unit was to shut down all non-essential services in the server room, to avoid generating extra heat. Lights included. The two doors between the server room and the IT area were to be opened (the rest of the building being on separate AC), and two enormous fans were to be placed in the doors and turned on to exhaust the server room. The on-the-wall thermometer was to be monitored closely, and if it rose above 72 degrees, the operator was to send system messages to all users and announce on the office intercom that the server was going down, and begin the rather lengthy shut down process on the machine. This was all documented in a big black binder of “Emergency Procedures”, which also included operations for floods, fires, and tornadoes as well as contingency plans for getting systems up and running at an alternate location in case the office were destroyed by a tactical nuclear strike.