MyPunchbowl experiences downtime (Sat morning)
This morning (Saturday), our internet hosting provider experienced significant downtime. For about 4 hours, visitors to MyPunchbowl.com received a “We’re experiencing technical difficulties” message when trying to visit MyPunchbowl.com.
We apologize to any customers that were affected by this outage. We’re lucky that this happened on a Saturday morning (from around 2am-6am PST). Our site is back up and running smoothly now.
For the technical geeks out there, the following is an explanation of what happened from our hosting provider Engine Yard:
Early this morning on March 15th, at about 2:05 AM PST (GMT -8), we experienced a power disruption at our Sacramento data center.
At 2:05am PST, we started losing connectivity to cluster ey01.
Our engineers worked with the data center and determined that power to ey01 had been reduced or cut entirely. Keep in mind that clusters have 2 power connections, each capable of handling the load alone.
Other data center customers and infrastructure were affected - not just Engine Yard. But at least only 1 of our clusters was affected.
A full cluster reboot takes time, but not usually a long time. In this case, the problem was greatly magnified because the outage was not planned nor orderly, so we needed to do a cold reboot and make sure everything restarted in a good state, including integrity checks on all databases on ey01. Some websites were down as long as 4.5 hours, depending on which resources they needed. ey01 was fully back online by 6:35am Pacific.
Customers on other Engine Yard clusters were not affected.
We take this very seriously and are in contact with our data center to figure out exactly what happened and what measures they’re taking to prevent it from happening again.
We are very sorry to Engine Yard customers who were affected by this power outage. We will be working with you to process refunds for 10X the downtime experienced.
