RE: Statement of Downtime 27th February 9PM 28th February
Dear Sir/Madam,
As you are aware the past twenty four hours have been somewhat a nightmare for service availability.
You were informed yesterday we had to complete urgent MySQL / Database work in order to implement our new service; this work has now been successfully complete with 95% of accounts restored at 5PM GMT on 28th February 2013. As there has been no full explanation I would like to take the time to give you a full update down and understanding of the issues causing the actions required last night.
Prior to this week, we operated from two servers with one main webserver hosting our entire range of services, as well as a backup system as a contingency to take the load and store user backups. This method lead to a significant amount of stress on our servers with the latest growth so last week we implemented two possible strategies in reducing the load times and freeing up server space. Of which the most cost effective, and future proof with the allowance of now being able to add on additional slaves easily we opted for the method I will explain further on.
Yesterday afternoon we looked in work around options available to setting up MySQL to fully work as we had encountered problems whereby users were not matched with the MySQL tab on cPanel. It was decided that the best option would be upon the discovery of broken PERL modules, to turn off all services and make a full backup which should of taken no longer than 3 hours. We decided this was the most appropriate method, and by taking off services it reduces the probability of data redundancy. Following the first hour of backups we anticipated that from the per hour it would have taken 5 hours to complete the backup, but in the end actually taking 8 hours to complete.
Last night we turned off all services at 9PM to allow engineers to complete the backup and start working on the SQL issue, we decided with our backup to format the primary HDD of our webserver and transfer all files onto our new server and then run the cPanel backup from there with the intention to start 5AM based upon our calculations of times to restore.
This morning all services were resumed at 6AM and services were gradually being restored, expecting to take roughly between 4 6 hours for every site to come back online, a lot longer than we anticipated. During this time we experienced SQL Timeouts whereby large databases were not transferred and stalled the entire back up process.
By 10:30AM all main reseller accounts were back online and functioning, several PHP modules were missing but were reinstated by 12PM. Due to connectivity issues with ourselves and Virgin Media being unavailable due to area maintenance, we were required to restore backups via 3G Dongle. By 4PM 80% of services was back online, however some accounts had failed on the initial backup process so were put to the back of the queue once the initial backup had been completed. By 9:30PM 98% of sites were back online with a few left to go due to database issues which are in the process of being resolved.
Further Problems
If you are experiencing further problems please contact our Support Team via the Billing Panel and we will do our best to get back to you as soon as possible in order to resolve your issue.
Compensation
We always try to maintain a 99.9% uptime but feel the past 24 hours have been a challenge and we would like to apologise for the inconvenience caused. We understand the burden and affect it has and never envisaged such a problem to occur with our services. This move we expect to lead to greater availability as provide a faster service. We have today moved over to a 1000Mbit port and our own vLan which should isolate us from any network attacks on other customers within our network.
As an apology, I would like to issue you with a 20% discount coupon off your next bill and a 50% off coupon for any domain available on our website advertised price. If you have customers using your service, feel free to pass the voucher on to them If you are an affiliate, you can still receive commission for domain purchases.
Invoice Coupon1 SQLMOVE
Domain Coupon2 - 50MYSQL
Kind Regards,
James Ensor
James Ensor
Managing Director
5wire.co.uk
1 Invoice Coupon is valid only for existing customers until 31/03/2013.
2 - Domain Coupon is valid for new and existing customers until 31/03/2013
Thursday, February 28, 2013
