Hey everyone,
Thank you again for your patience during our outage and for waiting for this follow-up!
This post has the additional details we promised last week.
The site consists of a caching proxy, an application host, an S3-compatible object store, and a database-as-a-service instance, all at the same well-known cloud hosting provider.
On the day of the outage, the database and object storage services both failed. While the object storage service recovered fairly quickly, the database service never recovered. Instead, the cloud computing provider’s support forced us to do an export of the instance by hand and totally delete the instance from our account.
We decided to restore the export of the database to a normal VPS running MySQL we’ll manage ourselves, along with the other compute nodes (the proxy, the app/web nodes, IRC, etc.).
To say the least, we’re very unhappy with the response from the provider, their inability to restore from the last backup, what amounts to the total failure of their database-as-a-service product, and as such the delays in restoring the site.
We’re still thinking about what’s next for hosting the MLA. In the coming months, expect a post about what we’re planning to do and what options we have.
Thanks again for your time on this and for sticking with us. Special thanks go to the folks on the IRC channel for moral support!
Best,
Cory & Tyler
Thank you again for your patience during our outage and for waiting for this follow-up!
This post has the additional details we promised last week.
The site consists of a caching proxy, an application host, an S3-compatible object store, and a database-as-a-service instance, all at the same well-known cloud hosting provider.
On the day of the outage, the database and object storage services both failed. While the object storage service recovered fairly quickly, the database service never recovered. Instead, the cloud computing provider’s support forced us to do an export of the instance by hand and totally delete the instance from our account.
We decided to restore the export of the database to a normal VPS running MySQL we’ll manage ourselves, along with the other compute nodes (the proxy, the app/web nodes, IRC, etc.).
To say the least, we’re very unhappy with the response from the provider, their inability to restore from the last backup, what amounts to the total failure of their database-as-a-service product, and as such the delays in restoring the site.
We’re still thinking about what’s next for hosting the MLA. In the coming months, expect a post about what we’re planning to do and what options we have.
Thanks again for your time on this and for sticking with us. Special thanks go to the folks on the IRC channel for moral support!
Best,
Cory & Tyler
