Major site outage 8 January 2019

Posted on: Wed, 2019-01-09 - 19:27 By: danie

Last night was not a good night... just after I had finished putting the last touches to adding Disqus comments (I'm still trying to remember how I restricted them to only certain pages) I made the big error of disabling two aggregator feed content systems I was no longer using and then also deleted (I should have disabled and backed up) a "feed" item which I thought I was not using. Then it all fell apart as I'd made so many tweaks that the more I tried to re-add the feed item I broke my RSS importer and things went from bad to worse.

I made the call eventually at 4am to rather restore a backup from 24 hours ago but as luck has it that backup failed with an error and had broken the whole site. Try as I could I could not get the backup to restore and eventually logged a fault with my ISP at 05:24 to assist with the restore as even the cPanel restore only resulted in an empty database.

My ISP struggled during the day and eventually tried a full hosting restore (rolling all my sites back to 7th January) but it still left the Drupal with an empty database.  After shifts changed at the ISP I thought I'd have one more try to restore the Softaculous backup... and this time it just went through without errors and bang the site was back. All I can think is that there was some corruption that prevented the restore last night, and the other restores today had set that right.

So as I sit here now I've been awake for 33,5 hours... actually feeling good so will probably sleep well tonight at least!

Open post to Comment