This morning, there was a power failure in our California data center – Equinix – taking all Zoho services down. As our services went down, it down all services, this means it caused database inconsistencies.
Services have multiple database clusters – like 8-12 database clusters per each service. Each cluster having a Master-Slave combination. Power failure caused inconsistencies in some of these clusters. But to restore these services fully, we need to make sure all these database clusters are consistent. This database consistency check and sync is what is taking time.
Realistically, it is going to take few hours to restore our services after making sure the databases are in sync. Some services will be restored sooner based on the level of inconsistency.
We sincerely apologize for this issue. Will keep you updated.
Update (1:35PM): Zoho Mail is partially restored. Users in affected clusters will be able to access it after we restore that cluster. For now, most Zoho Mail users should be able to access Zoho Mail.
Also, please note that Zoho Support is also Restored.
Update (1:40PM): Zoho Reports is now restored.
Update (1:48PM): Zoho Writer is now restored.
Update (2:25PM): Zoho Calendar & Contacts are restored. We still are working on fully restoring Writer. Will keep you posted.
Update (2:55PM): Zoho Writer is back up. Zoho Discussions & Zoho Books are restored.
Update (3:15PM): About 60% of database clusters have been restored currently for Zoho CRM. We will be making these available first. Users on these clusters will be able to access their data. As and when other clusters are restored, we will bring them online.
Update (3:50PM): Zoho CRM is partially restored. Around 80% of users should have access to their data. Other 20% will be restored soon. Will keep you updated.
Update (4:00PM): Zoho Invoice is restored.
So far, the following services are restored: Zoho CRM, Books, Invoice, Mail, Support, Reports, Discussions, Calendar & Writer are restored. Others will follow soon.
Update (4:20PM): Zoho Projects and Wiki are currently restored.
Update (4:28PM): Zoho Recruit & People are now restored.
Update (4:35PM): Zoho CRM is fully restored. Zoho Show & Bug Tracker are also restored. Zoho Docs is partially restored.
Update (4:45PM): Zoho Mobile service is now restored.
Update (4:52PM): Zoho Sheet & Docs are partially restored
Update (5:35PM): Zoho Chat & Share are currently restored.
Update (5:56PM): Zoho Creator is now restored.
Update (6:12PM): Zoho Meeting is now restored.
All Zoho Applications are currently restored. There are a few backend services currently being restored to full service.
We will do a detailed blog post with the postmortem report. Again, apologize for the outage.