UPDATE: Dec 4, 0800 Pacific Time
Mail has been normal over the last several hours. Messages are getting delivered immediately, with normal response times. The backlogged messages appear to have cleared completely. We waited until the US east coast customers started their day to make sure no new bottlenecks sprung up. So far we have none. We will continue to watch carefully. Once again, thanks for the immense patience from all of you.
A CLARIFICATION: Dec 3 1800 Pacific Time
We have received many questions and enquiries on whether the Mail issues we face are a result of not having geographically dispersed, redundant systems within the Zoho infrastructure. Here’s one customer opinion:
“Clearly, there is a need for redundant systems in other geographical areas that would be far enough away to not be impacted by any type of disaster that might affect the main site, and the switch-over should be quick and transparent. I’ve worked in IT for 32 years for various world-class IT providers, and all of them designed their systems with automatic redundancies and both automated and manual monitoring around the clock.”
The recent issue with Zoho Mail is not a result of our systems being impacted by the Chennai flood disaster. That would be a misunderstanding of the situation. In fact, the mail servers and storage systems are not located in Chennai. Zoho has multiple, redundant data centers across multiple continents. We have six copies of all customer data, replicated across data centers.
The Mail problem coincided with the natural disaster and was not caused by it. As explained in , the Mail issue was a result of a software bug that had unusually severe service consequences, even after it was fixed. It was our complete bad luck that this bug showed up at the same time as the disaster. So, no, we are not using the natural disaster to mask another underlying cause.
What we did say was that our ability to respond to the bug – examine it, test its scope and impact, fix it, test the fix, and update the software accordingly – was certainly impacted by the scarce availability of the technical staff of the Mail development team. This team is based in Chennai and reeling from the floods. In addition, our ability to respond to this bug’s unique aftereffects on the customer experience required the technical expertise of the same team. No additional system redundancy could have fixed this issue. The people with the specific skills were simply up against a bigger struggle.
Think of it this way. If an automaker uncovers a new issue with the braking system on one of their car models, the issue must be examined and resolved by their engineering team at their research headquarters. The extensive service infrastructure available at their hundreds of dealers cannot solve this problem. The technical expertise – and all the equipment for its investigation and resolution – are only available with the R&D team, a core centralized corporate asset. Now if that R&D team in unavailable for any reason, the issue remains unresolved until they return, and the customer unfortunately suffers.
That’s exactly what happened here and we’re really sorry about it.
UPDATE: Dec 3 1200 Pacific Time
We have identified the issues with our Mail – both incoming and outgoing messages – and are working on it. We have posted some answers to your questions here. All other services are up and running normally.
UPDATE: Dec 3
Unfortunately, our Chennai office has completely shut down. This is a sad first for us in over twenty years of operations. We had four redundant Internet connections and the last one shut down early this morning. The city is a disaster zone. This has been the worst rainstorm and floods in almost a hundred years. The airport is closed and most rail connections are suspended or diverted. Thankfully, our employees and their families are safe. We had a few cases that required rescues from flood waters and abandoned apartments. Other Zoho employees were among the first to respond. The city continues to suffer. Incessant rain is part of the forecast for the next few days.
We expect that Chennai will take some time, perhaps weeks, to recover completely. We are planning accordingly. We have opened up customer support centers in Austin (USA), in Tenkasi (India), and have moved 50 available technical staff to Bangalore to set up another emergency customer service and support center. More people will follow. We will be coordinating across all these offices to try and provide the best support we can over this period. Service will be slow. More importantly, problems will be difficult to track and resolve because many of the skilled and technical staff, along with their families, are victims of the floods themselves. Some have had to evacuate their homes. Many are unreachable today.
All Zoho services continue to be up and running as they are served from our distributed data centers worldwide. It’s the people-related issues that have been impacted, primarily customer support and issue investigation and resolution.
As of now, Zoho Mail is not working reliably, at least for many. Customers are experiencing problems receiving mail. We had been making progress on this issue over the last 24 hours and were testing out a fix, but the office shutdown has set us back. We are scrambling now to assemble the staff and location for a dedicated emergency Mail team right away, to pick up where we left off.
We will continue to post updates here and on Twitter. We completely understand that your businesses are also hurting and that many of your customers are impacted. We are truly sorry. We are focused on fixing this now. The vast majority of you have extended your goodwill and patience to us and we sincerely thank you. For the second time in one unfortunate month, we request you to stand by us.
Dec 1, 2015
Some of you have noticed and complained that our customer support has become less responsive over the last few days. Unfortunately, this is indeed true. Our support is completely backlogged because of the flood disaster in Chennai.
Many of you might know that Chennai is suffering its heaviest rains and floods of the last decade. They have battered the city for more than a month now and the forecast today predicts a worse downpour over the next four days. More than 10,000 residents have evacuated, including some of our own employees, and many are unable to get to work due to severed rail and road connections.
Zoho has increased support from our offices in Austin, USA to serve global customers during this emergency. People are also working from their homes, where possible. All this will certainly help but we still expect delays in customer service. Please be patient with us.