What a mess!

Should be fixed now ... basically the posts table and it's 1.5 million records crashed, had an index error and wasn't recovering properly. Spent all yesterday working on it and a process I ran overnight seems to have fixed it, thank goodness.

Sorry for the downtime.

Did torrino bring you coffee?

Sounds like sabotage. I wonder who has sufficient admin privileges to pull such a stunt...

Nah, nothing like that. Looks like the DB crashed partly through an insert, and because the record was partly inserted, indexes were out of wack and everything went bonkers.

That really sucks, I really appreciate your work. I am sorry you had to spend so much time fixing it though.

Thanks for all your work to fix it!

What the hell does that even mean?

Thanks for all the (unexpected) hours of working through partially inserted indexes and whatnot.

Heh, I'll try and explain better for anyone interested ... basically ... each post has a unique number:

Post 1, Post 2, Post 3, etc. When someone makes a post, the database will automatically give that post the next unique number. Unfortunatly, something happened in the middle of an insert, meaning a record was entered into the database as Post 4, but the database got amnesia and doesn't remember inserting it. Well, the next time someone tried to post, the database again tried to insert the new post as Post 4 and was like "Hey, who the hell are you! There's not supposed to be a Post 4 here!"

That's the basic issue, and fixing that issue isn't as simple as you'd expect. Each time a post is made, the posts table is LOCKED temporarily, and UNLOCKED when the insert completes. Well, since it never finished inserting correctly, the table remained LOCKED, which causes all kinds of headaches. All attempts to unlock the table resulted in the fail whale.

The only way to fix these two problems is a series of repairs on the database, but because the table is so huge (~1.5 million records), attempts to repair it were causing the HTTP service to crash. To get around that, I had to run the repair via command line, but part of the repair process is to build a temporary index file which was too big for the directory it builds in. Meaning it ran out of space and couldn't fully rebuild the database. To give the specific partition more space, I'd have to reformat the entire server, and I didn't want to do that. So I then decided to rebuild the index in memory, which was very SLOW, but seemed to be working. Unfortunatly, when that process finished, it threw an error saying the database was still marked as crashed, and rebuilding the index wasn't sufficient to repair it. Yeesh!!!

So I had to rebuild not only the table indexes, but also the file holding all the data of the database. I kicked that process off before bed, it finished early this morning. Ta Da!

Bet you wish you hadn't asked, heh.

Thanks Karl-izzle!

Thanks, Eagle

Thx Karl
I luv this place n don't ever want it to die.

Yay it's back

Thanks for your hard work fixing this Eagle!

Thanks brosef

Muchas gracias! I missed my KJ lurking yesterday and with all the threads about the site disappearing I was a bit worried. great that it's back up.

Yeah I thought it's over now, I'm really glad it's not.

Thanks Eagle! No matter how much guff you get sometimes from certain posters, I've always admired your hard work.

Seconded.

Yeah, Karl, thanks a lot!

Thank you very much, Eagle. I will dedicate my first daughter to you.

Hmmm.. Must've crashed from the AVATAR trailer Premiere and it's overwhelming popularity..