Nearly a month ago now I got a call early in the morning. It was from our IT lead. The Datacenter...was gone.
At first I thought we were under some sort of DOS attack. Then I feared the worst, some sort of Ransomware? Were we going to find out that everything was encrypted and gone? We soon learned, no—but in some ways, it was worse. Everything we had at the Datacenter was wiped out. Everything. 34 Terabytes spread across a dozen+ servers of various ages with unique configurations dating back decades. Some of this stuff had been brought over from OS/2 back in the day. Some of the configurations were originally set up on NT 4 for PowerPC! Gone.
We have near real-time backups to dedicated servers at the datacenter but they were wiped out too. For legal reasons, I can't go into anything more specific than to say everything was gone.
We also do nightly backups off-site as well. But these are the kinds of backups you don't really expect to ever have to use. They're the equivalent of the old tape drives we used to use back in the day. You back up to them but you don't really anticipate ever needing them. But here we were.
The backup was a single, 34TB file. We debated how to get the data back fastest. I mean, couldn't we just put them on a removable HD and ship it? Well, not so much. The NAS or whatever it used at the off-site service has a USB 3.0 connection. Getting the data off would take many days and then have to be reuploaded via the USB HD.
Ok, so transferring it is. We have a 10Gb/sec net connection, how long could it take? The math made it out that we should be able to get it in just a few days. Nope. Weeks. It took weeks.
And once we got it back, things didn't look good. While the raw data was there, it was not obvious how all this would hook up again. Remember, we are talking a hodgepodge of servers that have been around decades. You can't just spin this stuff up.
I talked to Jafo and LightStar and others and I told them I didn't think we'd be getting WinCustomize back. Even if we got the skins and themes back, the spaghetti of the skin library databases would not so easily be brought back to life. Even now, it's not clear how well connected the various files will be.
Near the end of March I wrote:
Dear Artists, Skinners and Designers,
For 24 years we have created amazing things together. I am sad report that we suffered a catastrophic data loss. This data loss also consumed all of our on-site backups as well. While we did have many offsite backups, the sheer size, complexity and fragility of the data has made it impractical to restore them. As a result, we have made the painful decision to close WinCustomize.com.
We have enjoyed the near quarter-century of collaboration, conversation and creation with you. I am sorry to see it end. It is, indeed, an end to an era.
Farewell,
-Frogboy
But the team didn't want to give up. Pat, myself, Jillian, and others from the 2001 era are all still at Stardock and the team was desperate to save WinCustomize.
And so tonight, as I write this, we're...almost back. The site definitely is worse for wear. We are going to need your help to rehabilitate the site and we will be grateful for as much grace and patience that can be sent our way as we still struggle to bring it back. But I am happy to report that WinCustomize has survived—and we couldn't have done it without the incredible determination of our team and the loyalty of our community. It's going to be a journey, but it's one we're committed to taking together.
