Hardware issue with UK MySQL Server Phobos - Powered by Kayako Help Desk Software

News

Jun 26	Hardware issue with UK MySQL Server Phobos Posted on 26 June 2010 09:31 AM
We are currently investigating a hardware issue with Phobos, the UK MySQL server which also hosts outgoing email from websites. Once we have more information we will post it here. UPDATE: The problem is with key hardware components in the server. We are currently pursuing two different options at once. The first is to attempt to use spare hardware to get the server back online. In case this does not work, we are also preparing one of the spare servers to take it's place and will restore all databases from this mornings backup onto it. UPDATE 2: The on-site technicians were unable to bring the server back online. As such, we are proceeding with the second option that was begun at the same time. A replacement server has been set up and we are currently restoring all the data onto it. This process is unfortunately likely to take several more hours. The backups being restored are from approximately 8am this morning, so very little data will have been lost. UPDATE 3: The restore is still in progress. As Phobos also is responsible for sending emails from scripts on websites, these emails are currently being delayed until the server is fixed. Incoming email and normal outgoing email (which as hosted on Galatea) are unaffected. UPDATE 4: The majority of MySQL databases are now online and running. We are working to restore the last few remaining. UPDATE 5: E-Mails sent from websites are now coming through properly. There is a backlog to process, and they will come through over the next hour or so. FINAL UPDATE: All MySQL databases and outgoing emails should not be working fine. If you are continuing to see any problems with any of your services, please contact us immediately so that we can look into it. POST-ISSUE UPDATE: We have had reports that during the restore, Helm has set all MySQL usernames to be lowercase. If you are still having connection issues, please try setting your username to lowercase as this may fix any issues.