The worst case scenario for any business is a total server and local backup loss with nothing to restore to. Find out below how we dealt with this extreme scenario and what impact this had on the end user.
A bespoke engineering company of 332 users, their company produces approximately 9000 emails per day. Due to budget restraints the server was poorly maintained, 7 years old and not fit for purpose, the database was 431GB and the server barely hanging on but email was successfully sent and delivered most of the time.
Mitol PerfectBackup had been employed to back up the server at database and mail level. Onsite backed up to a NAS device twice a day, offsite ran every night to our datacentres. All backups had run successfully for some 2 years and a 90 day retention period had been set. All looking good as long as the server holds on.
Mitol received a call early Saturday morning stating the server had died, not because of age, but because of the burst water pipe above the server Unfortunately, by the time the client found out their pipe had burst the water ingress had also killed their local backup NAS device. It was at this point we found out there was no redundant hardware or OS or application media of any type on site.
To ensure all 332 users have full access to email, contacts, calendars and public folders by 7am Monday morning. In other words, staff who left for the weekend would not know of any issue when they arrive back at work 7am Monday morning.
After the client finished his brief scream of panic the first phase of our availability plan was to move all 332 users to a temporary hosted exchange solution. This would give breathing space for phase 2, a full recovery of the 2003 Exchange database whilst upgrading to latest version 2013.
A restore of client's .EDB and .STM was started as client started to create a list of email accounts to be set up. Client edited their MX records to point to our infrastructure this would ensure no incoming email would be lost. Upon receipt of client’s full list of email addresses we then started to add them to our hosted exchange environment.
Three hours after initial disaster call, the client's exchange databases had been recovered and full list of email addresses had been supplied. We were now in a position to start exporting PST's from the clients database and adding all mailboxes to our hosted exchange infrastructure. Whilst we carried out "data centre tasks" the client had already started creating new Outlook profiles on user's computers.
Six hours after initial disaster call. The client had created 332 new outlook profiles and his involvement in phase one had now finished. Meanwhile our support staff had finished creating mail accounts and continued to import PST's as they were extracted. As PST's were imported an automatic synchronisation started with users Outlook client.
Eight hours after initial disaster call. All PST's extracted and synchronisation process is in full flow. We are lucky the client has a 20MB download available. Key point is that from an inbound point of view, the client has only suffered one hour of downtime. Since the mx record was changed no inbound emails were bounced. So far a lot of manual intervention has taken place, but given the circumstance, no hardware to restore to, no software or media available things are as good as it gets.
Thirteen hours after initial disaster call. All data restored and synchronisation complete. As of Sunday morning all 332 mailboxes are fully operational, spam filters fully operational and one IT Director very happy.
Monday Morning. All users apart from one didn't notice any changes. The only comment made was "did someone delate my spam already?". Phase one, mission accomplished! As you can imagine the IT Director spent a long time with fellow directors discussing how his weekend was and how they collectively ensure such a data loss scenario does not happen again and how Exchange is brought back onsite. Three months of onsite high availability Exchange quotations ensued. The quotations included multi server DAG environments, stretched clusters and other high availability offerings, initial prices for hardware, software and installation ranged from £22,000 to £35,000, there were then ongoing costs of up to £900 per month for support maintenance and management.
Whilst the monthly cost of our hosted exchange solution for 332 users worked out marginally more than if the onsite deal was financed over three years the client decided to stay with our hosted email solution. Because the client now understands the importance of their email along with benefits such as improved uptime, and fully managed maintenance and security, which in turn help employees to be more productive and IT staff to be more effective the client it proved not to be a difficult decision.
Accountancy firm was embarking on a server refresh. As a Windows 2003 to 2012 OS upgrade was also planned the location of critical applications and data would be different. It was our job to ensure online backup service continued as normal without requirement of a fresh, full 400GB upload.
The end users IT reseller finished the migration, tested and all looked good. Mitol support was then allowed shared remote access where we recorded changed data locations.
Once all changes were documented and agreed, we then restored client data to a local server at one of our data centres, created a new virtual server as per a mirror of client’s new server and transferred data so everything looked identical. We then created a new backup set and ran the backup process from our local virtual server to our backup infrastructure. This was technically the large upload but it did not originate from the client's server.
Once complete we transferred all backup settings to the client's new server therefore negating the need to do a first large upload from client server, and the backups ran at an incremental level there onwards.
The client and reseller was so happy with our work we received the email below.
To the MD at Mitol,
We have been a customer of yours for some years and have found your service from the IT guys outstanding. We deal with Graham and Jacob in particular but have found any of the IT support team so helpful, polite and very knowledgeable.
We have had a new server installed this week and Graham has liaised with our IT guys to ensure the smooth install and running of the backup systems.
There was a small problem today which Graham resolved very quickly. He came up with the solution immediately and our back up is now up and running again.
I hear he even stayed in the office until 8pm to ensure our 1st backup on the new server was working. This is really commendable and I thank him personally for all the help and assistance.
In addition to this our IT guys, in Maldon are very impressed by the support given to them during the install of the program onto the new server.
As a complete 'non techie' your guys are fantastic. I have recommended your product in the past and will continue to do so. Please relay my warmest thanks to Graham for his assistance and outstanding work over the past week and to Jacob for his help to us in the past.
Please ensure this communication is forwarded to your MD and Manager please and pass my thanks on to your team.