As many of you witnessed, Sylestia experienced many issues Wednesday afternoon that led to the eventual 24-hour maintenance that we've been performing on the Site. I will try to explain everything as best as I can below:

So, What Happened?
At some time between 3:41pm and 3:56pm on 3/25, our database began experiencing some crashing issues. While it was not a complete crash (which would have been better lol), it was only a partial crash where only a handful of our tables crashed. This ended up causing a lot of issues.

Firstly, any page or interaction that attempted to access these tables resulted in an error as a crashed table is inaccessible. This is why the Site was still "working", but many things were not. Because the tables that were fine were still working as normal, but the tables that were crashed were inaccessible.

The second, more important issue is that when entries on these tables were attempted to be accessed or modified, it was causing them to become corrupt. Those corrupted entries were then deleted from the table.

Unfortunately, Fai and I were not at home when this occurred and did not even become aware of the issue until around 8pm when we took the Site offline. This meant that the crashed tables had about 4 hours to cause all sorts of complications with the Site.

As to what specifically caused this to happen, it's hard to say. Our host and I have spent the past 24 hours checking all sorts of things to make sure everything's intact and fully functional. We did not find any issues. Also, simply running a repair operation corrected the crashed tables. So most likely this was just caused by some sort of glitch. Servers are computers and run just like any other device. And sometimes even the best of devices can have a glitch. A Site's server is an incredibly complicated machine with many moving parts. Just one glitch can have catastrophic results if it's in the wrong place. And since our server is processing hundreds of data entries every few seconds, it can cause all sorts of issues.

Wait, Lost Data?
Yes, some data was lost. Fortunately, we run a backup of our database every 3 hours. Through some work, I was able to figure out the specific entries that were missing and restore them to what they were at our latest backup (which was at 4:05pm). The following is what I know was lost (and restored):

-726 pets were lost, but all were restored
-1 stable was lost, but was restored
-27 items were lost, but all were restored
-some other information data was lost, but it's not vital to anything

There may be some other data (besides what I will describe below) that was also lost that I am unaware of. If so, please just let me know and I should be able to restore it by looking at our backup.

The Bad News
The bad news is that this unfortunately went on for about 4 hours. During this time, many transactions were taking place. If a transaction occurred that required one of the crashed tables, this transaction would not have processed correctly. Additionally, for the most part, I can't figure out exactly what wasn't processed because it was never actually created. As an example, if you had purchased traps from a shop during this time, the game would have charged you for your purchase but the traps never would have been created or added to your inventory because the items table had crashed.

So most likely, this affected many of you and I will have to manually correct these situations for you.

Therefore, if you are missing ANYTHING from your account such as traps, potions, philters, etc. Or if you attempted to generate a pet... buy a stable, etc, etc. Please just send me a PM and provide as many details as possible. I will have to look through our database, our backup, and then figure out what happened and what I can do to fix it for you.

Additionally, if you find any bugs or constant warnings, please let me know as well so that I can correct them.


Thank you to everyone for your patience during this downtime and we apologize for the inconvenience. Fai and I are unfortunately more than aware of the frustration that sets in when a game you want to play is offline. So thank you!

Did the problem stem from hardware? say a drive head crash? maby quantum tunneling?
SYLESTIA LIVEESSS!!!!!!

Link: https://www.sylestia.com/forums/?thread=23758&page=1#2 Author: Fennectech Time Posted: 3/26/2015 at 8:50 PM Did the problem stem from hardware? say a drive head crash? maby quantum tunneling? SYLESTIA LIVEESSS!!!!!!

No, the hardware all checked out.

Right on! Thank you!

so no real idea where it could have come from? every glitch has a source maby ommnomnom took a bite of the tables?

Glad everything was fixed :) I honestly wouldn't even realize probably if something was missing, but I wasn't on during the issue times much. I just came and nurtured my eggs and left. Glad you guys were able to fix the site :)

Can I just thank you for all your hard work and dedication, Krin? ROUND OF APPLAUSE FORT KRIN, EVERYONE!

Link: https://www.sylestia.com/forums/?thread=23758&page=1#5 Author: Fennectech Time Posted: 3/26/2015 at 8:51 PM so no real idea where it could have come from? every glitch has a source maby ommnomnom took a bite of the tables?

Our host's best guess was just un-optimized mysql settings and it got tripped up. Since we weren't here, it's really hard to see any closer than that. Coulda just been a bad query that ran, a really long query that got stuck, etc.

We ran a bunch of checks on everything, tuned some settings, and will just hope for the best. =P

The crashed parts of the site confused me a bit, but I'm so glad all is in order now. Thanks for being such great Admins!

i did see several tables go wrong on my own sql server (i use it to log minecraft grief) ended up erasing the tables as they were not mission critical