Hi,
It's been a long time, but we're finally back. Let's recap WTF happened.
On the morning of September 15th, our web host (Hyper Expert) had an outage with a number of their servers located in Seattle. WWEF was one of the many websites impacted by this. According to their Incident History, this outage lasted a total of one hour and twelve minutes. These kinds of things happen every once in a while, certainly no big deal. I was working on third shift at the time, so I was awake and frequently refreshing the Service Status page following the updates. I ended up getting off work and getting home, only to see that they had updated their Service Status page claiming that everything was back to normal. Unfortunately, that wasn't the case, and that prompted me to open a support ticket. I'd guess that this had a 48 hour period of absolute nonsense going back and forth with their useless support team. Basically, I kept being told that this was my problem to solve, due to the fact that we are on an unmanaged VPS. While it was unmanaged, how can they make that statement when something happens on their end that adversely impacts my website(s)? That's where they need to take ownership and correct the problem. Malcolm was there every step of the way helping getting that figured out, but we literally didn't even have the means to solve the issue. We were completely locked out of SSH. Unable to access our DirectAdmin backend, obviously unable to access WWEF itself.
After going back and forth with their support team, the gentleman that I'd say owns Hyper Expert was able to correct something on their side to get it so we could at least log back in. And that's where the work began. We got the site back up and running. We were pissed about it, but we were scrambling to keep this downtime minimized to 2-3 days. After going through this ordeal, the gentleman said "Hey, I'm sorry for your troubles. Here's a few months of free hosting to make you feel better." That's cool, I guess? But why would I want to be with a web host that was absolutely useless when it mattered most anyways? I don't know. Anyways, it gets worse. This guy went and changed my invoice date to reflect this credit of free hosting that I was receiving. He changed the invoice due date from September 27th to January 27th. That's four months of free web hosting. Cool, right? Nope. Instead of changing the invoice due date to January 27th, 2021, he changed it to January 27th, 2020. May not seem like a big deal, but it was. At midnight, their billing system then saw that I had an invoice due from January 2020 that hadn't been paid, which was many months ago, obviously. Their automated system not only suspended my VPS, but it also completely discarded our server entirely. All because of a typo.
Then the game of cat and mouse began with their support team once more. Received very little assistance this time around, other than being told, "Your server is back online, sorry about that. Here's two more free months of hosting to make you feel better." Mind you, they brought the server back online, but they literally just did a fresh server install, as they had totally erased our server, as mentioned before. Support again left it up to myself and Malcolm to resolve. But yeah, I'm just like "Okay... Fuck you, Hyper Expert." After that situation, we decided we were going to change hosts. We decided to cross the border, and Web Hosting Canada is now hosting our VPS. If you have ever owned a website, the one thing that is emphasized more than anything else is the importance of having backups. That's the difference between life and death for a website for obvious reasons. You never know when a server could randomly explode somewhere in Seattle or somewhere, and all of your shit gets lost. Backups, backups, backups... We had weekly backups. Unfortunately, the SQL files attached to our backups were all corrupt dating back for many months. That was uncovered while we were looking to quickly bring WWEF into our new web hosting environment. Literally every backup I had on my end was unusable. I opened a ticket over at XenForo.com regarding this, and there wasn't a lot of progress there either. There were able to identify the cause of the issue, but they weren't able to solve the problem. Basically, a particular add-on that I had installed soon after I took over WWEF is what caused all of these database files to be corrupt, and this is an add-on that I've been using for years. I then contacted that add-on developer, and hoped for the best. Received little assistance there, especially with there being a bit of a language barrier between us. So, what was next? I didn't know at that point.
With just a slither of hope, I messaged Solidus on Discord just asking if he happened to still have a backup file of WWEF from back before I came around. Fortunately, he did. I provided him with the SSH information so he could upload the most recent clean backup that he had. He uploaded it, but then when we went to set it up about 12 hours later, the file was no where to be found. Somehow someone with an IP address in India had logged into the backend of our server and deleted a few different files, including the SQL file that he had uploaded. Like, what the fuck? Apparently, another individual had been logging in and out of my Discord account for some time, due to me having a shitty password attached to it from many years ago. By that point, we had pretty much thrown in the towel. Solidus had told me that he deleted the copy he had right after he uploaded it for us, so I thought we were done for at that point. Then sure enough, maybe like a week later, Solidus was able to find another copy of the forum from somewhere, thus saving us. Thank god for Solidus.
Obviously, this whole situation isn't ideal. We literally uploaded a backup from February 1st. It is October. That means we lost the last eight months of data. We had new users sign up in that time span. We had some old users return. Plenty occurs in an eight month period. On a personal level, I had right around 2,000 posts from trolling the Jeffster in our Raw/NXT/SmackDown live chats. Now I have 15 posts. That's a ton of content gone forever, and I'm really sorry for that guys. From start to finish, this has been a horrible situation. It just seems like everything that could have went wrong... Did go wrong. We're going to do whatever we can to pick up the pieces and move forward though. As of today, I'd like to introduce Malcolm as our new sexy Technical Administrator, and I guess I need to announce Jacob's promotion to Administrator again as well. Anyways, Malcolm has been hard at work behind the scenes getting this place put back together. If you saw how broken the forum was last night, you can definitely tell that a lot has happened in the last 24 hours. Malcolm has upgraded our forum to XenForo 2.2, which was just released a few days ago. You'll see some aesthetic changes, like when you're creating a topic. I think it looks cleaner. I dig it. Malcolm also updated our live chat add-on and got that new and improved.
Anyways, I don't know what else to say. This whole situation was terrible. Welcome back guys. I've missed you all.
It's been a long time, but we're finally back. Let's recap WTF happened.
On the morning of September 15th, our web host (Hyper Expert) had an outage with a number of their servers located in Seattle. WWEF was one of the many websites impacted by this. According to their Incident History, this outage lasted a total of one hour and twelve minutes. These kinds of things happen every once in a while, certainly no big deal. I was working on third shift at the time, so I was awake and frequently refreshing the Service Status page following the updates. I ended up getting off work and getting home, only to see that they had updated their Service Status page claiming that everything was back to normal. Unfortunately, that wasn't the case, and that prompted me to open a support ticket. I'd guess that this had a 48 hour period of absolute nonsense going back and forth with their useless support team. Basically, I kept being told that this was my problem to solve, due to the fact that we are on an unmanaged VPS. While it was unmanaged, how can they make that statement when something happens on their end that adversely impacts my website(s)? That's where they need to take ownership and correct the problem. Malcolm was there every step of the way helping getting that figured out, but we literally didn't even have the means to solve the issue. We were completely locked out of SSH. Unable to access our DirectAdmin backend, obviously unable to access WWEF itself.
After going back and forth with their support team, the gentleman that I'd say owns Hyper Expert was able to correct something on their side to get it so we could at least log back in. And that's where the work began. We got the site back up and running. We were pissed about it, but we were scrambling to keep this downtime minimized to 2-3 days. After going through this ordeal, the gentleman said "Hey, I'm sorry for your troubles. Here's a few months of free hosting to make you feel better." That's cool, I guess? But why would I want to be with a web host that was absolutely useless when it mattered most anyways? I don't know. Anyways, it gets worse. This guy went and changed my invoice date to reflect this credit of free hosting that I was receiving. He changed the invoice due date from September 27th to January 27th. That's four months of free web hosting. Cool, right? Nope. Instead of changing the invoice due date to January 27th, 2021, he changed it to January 27th, 2020. May not seem like a big deal, but it was. At midnight, their billing system then saw that I had an invoice due from January 2020 that hadn't been paid, which was many months ago, obviously. Their automated system not only suspended my VPS, but it also completely discarded our server entirely. All because of a typo.
Then the game of cat and mouse began with their support team once more. Received very little assistance this time around, other than being told, "Your server is back online, sorry about that. Here's two more free months of hosting to make you feel better." Mind you, they brought the server back online, but they literally just did a fresh server install, as they had totally erased our server, as mentioned before. Support again left it up to myself and Malcolm to resolve. But yeah, I'm just like "Okay... Fuck you, Hyper Expert." After that situation, we decided we were going to change hosts. We decided to cross the border, and Web Hosting Canada is now hosting our VPS. If you have ever owned a website, the one thing that is emphasized more than anything else is the importance of having backups. That's the difference between life and death for a website for obvious reasons. You never know when a server could randomly explode somewhere in Seattle or somewhere, and all of your shit gets lost. Backups, backups, backups... We had weekly backups. Unfortunately, the SQL files attached to our backups were all corrupt dating back for many months. That was uncovered while we were looking to quickly bring WWEF into our new web hosting environment. Literally every backup I had on my end was unusable. I opened a ticket over at XenForo.com regarding this, and there wasn't a lot of progress there either. There were able to identify the cause of the issue, but they weren't able to solve the problem. Basically, a particular add-on that I had installed soon after I took over WWEF is what caused all of these database files to be corrupt, and this is an add-on that I've been using for years. I then contacted that add-on developer, and hoped for the best. Received little assistance there, especially with there being a bit of a language barrier between us. So, what was next? I didn't know at that point.
With just a slither of hope, I messaged Solidus on Discord just asking if he happened to still have a backup file of WWEF from back before I came around. Fortunately, he did. I provided him with the SSH information so he could upload the most recent clean backup that he had. He uploaded it, but then when we went to set it up about 12 hours later, the file was no where to be found. Somehow someone with an IP address in India had logged into the backend of our server and deleted a few different files, including the SQL file that he had uploaded. Like, what the fuck? Apparently, another individual had been logging in and out of my Discord account for some time, due to me having a shitty password attached to it from many years ago. By that point, we had pretty much thrown in the towel. Solidus had told me that he deleted the copy he had right after he uploaded it for us, so I thought we were done for at that point. Then sure enough, maybe like a week later, Solidus was able to find another copy of the forum from somewhere, thus saving us. Thank god for Solidus.
Obviously, this whole situation isn't ideal. We literally uploaded a backup from February 1st. It is October. That means we lost the last eight months of data. We had new users sign up in that time span. We had some old users return. Plenty occurs in an eight month period. On a personal level, I had right around 2,000 posts from trolling the Jeffster in our Raw/NXT/SmackDown live chats. Now I have 15 posts. That's a ton of content gone forever, and I'm really sorry for that guys. From start to finish, this has been a horrible situation. It just seems like everything that could have went wrong... Did go wrong. We're going to do whatever we can to pick up the pieces and move forward though. As of today, I'd like to introduce Malcolm as our new sexy Technical Administrator, and I guess I need to announce Jacob's promotion to Administrator again as well. Anyways, Malcolm has been hard at work behind the scenes getting this place put back together. If you saw how broken the forum was last night, you can definitely tell that a lot has happened in the last 24 hours. Malcolm has upgraded our forum to XenForo 2.2, which was just released a few days ago. You'll see some aesthetic changes, like when you're creating a topic. I think it looks cleaner. I dig it. Malcolm also updated our live chat add-on and got that new and improved.
Anyways, I don't know what else to say. This whole situation was terrible. Welcome back guys. I've missed you all.
