Database Shitfest

The title says it all; it's the best way to describe what is going on at school lately. We have an enterprise-esque database system at the center of our school. This DB has its fingers in everything: scheduling, transcripts, grades, attendance, admissions, general ledger and all sorts of other stuff. It was (and still is) functioning fine, but is slightly out of date. We stopped receiving updates more than a year ago and nobody noticed. Then this summer we discover we are at version 803 when the current version is 816.

Normally this wouldn't be a big deal, but I'm having a hell of a time trying to get up to date. Not only are we trying to update the actual data and the programs accessing it, the server hardware itself will be replaced once the update is successful. One of the problems is the fact that this is a production database and is in use about 101% of the working day. Sure I can kick people out of it for a while, but a few people's jobs revolve around accessing this database so I can't keep them out for any length of time.

What I've been doing is kicking people out, copying the data to the new server and then running the updates on the new hardware. This way I can put the old server back in action if there are problems upgrading. The first time the update failed about 10 minutes after I started. It started barking about some missing column in some table. I called the support line and they had me send them a copy of the database so they can figure out the problem.

A couple days later they get back to me saying they fixed the problem. Today one of the support people successfully updated the old data I copied over the first time. So I try running the update on a new copy of the current database. Guess what happens? First I ran out of disk space on the drive that the data was held on. There were so many changes to the data to bring it up 13 versions it created thousands of log files (to keep track of transactions), each being around 1mb, and filled the disk up. I noticed this when I went back to work around 5:30pm today to check on the update progress. I made some more room and tried the update again (3rd time). Yup, failed again. At least this time it was in a different place and I was able to get the error messages. I forwarded them to the support people along with another copy of the database.

This update has been a real pain in the butt. I have to work around about 6 people's schedules, wait for the support people to figure out the problems and find time to work with them. And this is all on top of the 10000 other things that I have come up since the middle of August and need to be done by the start of school (8 days).