2025-10-07 through 2025-10-09: Site Outage Update

[defor]

As many of you noted over the last few days, the MLA was unexpectedly offline over the last few days.

There was a hardware failure on the part of the provider that we expected to be resolved quickly, but there were delays out of our control.

As this was only a simple outage, and not a complete loss, we decided not implement full disaster recovery plans.

Many of you noted that we were unable to provide public updates as the outage extended, and for this, we do offer our sincere apologies, and we are currently revising our incident response plan and capabilities to ensure public facing communication is possible.

Thank you for your patience with this!

 

[wthww]

Hello all,

I wanted to post a quick post-mortem:

1) We moved the site in a prior maintenance period to some beefier hardware with additional storage during our last maintenance period to accommodate for media growth tipping the needle on making object storage viable from a cost and cost of backup perspective.
2) The leased machine it runs on experienced a motherboard failure and we engaged remote hands almost immediately after the site went down. Once remote hands determined it was a no-post situation, we went into the repair queue which required we wait a few hours.
3) Once it was our turn in the repair queue they pretty quickly determined the motherboard failed. This required they decide if they were ordering a new one or swapping us to a new machine. We opted for swapping the drives over to a new machine.
4) Once the drives were swapped over some minor reconfiguration was necessary to match up with the new hardware which brings us to the site being back!

Just as a reminder, the site has over 250g of media at this point. We intentionally don't overly restrict attachments or spin out old posts into a read-only archive to keep the entire site searchable and interactable. We've got 20 years of activity in active storage. The old girl is a bit too big to move around on the drop of a hat these days.

PS: Thanks to defor for running point while I handled the technical side of it! Appreciate you!

Thanks,
//wthww