Zoho services under criminal attack

UPDATE: 0800 pacific time, Nov 12

All Zoho services are now functioning  normally. We have performed many upgrades and are monitoring carefully. We'll make a detailed statement soon.

UPDATE: 1605 pacific time, Nov 10

Finally we have made progress with Mail! Mail should be clearing now and reaching you. The protections we put in place to mitigate DDoS attacks had the side-effect of affecting mail delivery. We have identified these and cleared them so things should be working now.

Please tweet us @zoho, or call support, if you're still not receiving mail. Your business runs on Mail and we can completely understand how stuck you must have felt. Thank you immensely for your patience and understanding.


UPDATE: 0915 pacific time, Nov 10

We continue to be on very high alert, watching all services very closely. The last several hours have been focused on performance issues and bottlenecks that many of you customers have reported over the last several hours.


UPDATE: 0615 pacific time, Nov 10

Due to the emergency rerouting - designed to mitigate the attacks - some links still have issues. As a result, you will see some slowness and delays on mail delivery. We're working on this and will keep you posted.


UPDATE: 1815 pacific time, Nov 9

The attacks continue and we are managing to stay afloat. We are not assuming that the worst is over yet and customers should not either. Stay with us.


UPDATE: 1545 pacific time, Nov 9

Almost all services have been up all day, today. Services are all much slower than usual, although we can see some improvement over the last few hours. Incoming mail is extremely slow.  Many customers have pointed this out and we're deeply sorry that it's directly affecting your businesses. Our team is working specifically on this issue right now.

Please read the other post in this blog, Answers to frequent questions, to get answers to specific issues. We will continue to update that as well.


UPDATE: 1034 pacific time, Nov 9

Most services are up, not all. Service continues to be slow. Working on it.


UPDATE: 0847 pacific time, Nov 9

Here is what is happening with our infrastructure. We had several pending updates that were designed to mitigate DDoS attacks. Those updates were scheduled to go live in the next 2-3 weeks. We have had to squeeze them in over the last weekend. That has created a bit of instability in our systems, given the emergency nature of the upgrades themselves.

So while we have taken successful counter-measures against DDoS, we are now working on strengthening our systems. We have traffic rerouted through our secondary data center and that introduces more hops that adds latency (delay) as well. We are working on all these issues right now. Thank you for your patience.


UPDATE: 0650 pacific time, Nov 9

Due to relentless attacks we have taken emergency counter measures. Part of this is to reroute traffic and data, through additional network hops to filter out the attack. This added complexity is making service access unstable and slow for customers. We are working on it as we speak.


UPDATE: 2015 pacific time, Nov 8

Services are up and we are mitigating the attack. We fully expect them to keep coming though and disruptions may yet occur.


UPDATE: 1705 pacific time, Nov 8

Zoho services were targeted again on November 8 at 1705 hours pacific time (0115 GMT). Services were down for about 33 minutes and came back up. Many are not reliably up yet. Our teams have been working nonstop over the last 72 hours with various counter measures. We are still at it and are prepared for tough days ahead. We expect more attacks and more service disruption as we get into the work week. Please stand with us at this time.


Zoho under distributed denial-of-service-attack:  Nov 6

Dear Customers and Well-wishers,

Zoho was subject to a criminal attack to our networks called a distributed denial-of-service attack (DDoS). This started at 8:15am pacific time on November 4, 2015 and has continued intermittently. The obvious intent is to make Zoho and all of our services unavailable to customers by flooding our servers with bogus requests from multiple sites. The attack was accompanied by threats and a blackmail attempt at extortion to prevent ongoing attacks.

This attack is focused on denying access since it targets the network connections to our servers. All your data are sound and secure, but unfortunately you cannot access it reliably. This is like a crowd of people standing around the entrance to your bank, preventing your entry. Your money is in there and safe, you just can’t get at it until we have them all moved.

Companies like us, that have offered services accessed through the internet for many years, carefully prepare for and expect these attacks. But the attacks are getting worse and more sophisticated. In fact, there have been other attacks this very week. Secure email providers like ProtonMail and Runbox have been hit. Many major banks and businesses with online customers have seen these attacks in the last year.

We’re working round the clock, with service providers, experts, and others to continuously improve our defenses. We are also contacting law enforcement. There is no single silver bullet to fix these issues and we have to work at it diligently and with the right experts behind us. We expect that these attacks will continue, but we also expect to prevail.

In the meantime, things will be rough for you, and you have our deepest apologies. The most unfortunate part is that we cannot tell you exactly when everything will be back to normal. We simply cannot know. We regretfully ask for your patience in advance, knowing fully well that you rely on us to run your business and to serve your own customers. We are deeply sorry for your trouble.

Please stand by us as we fight this attack. We cannot give in to criminals and embolden them to perpetuate other attacks. Thank you again for placing your trust in us.

Our status page shows available services at any time and we will communicate through this blog and through Twitter (@zoho).

Comments

43 Replies to Zoho services under criminal attack

  1. Unfortunately, I am not receiving nor can send any emails. I really like Zoho, and want to continue to use this services. I hope this issue is resolved soon.

  2. All working perfectly this morning! :) Fantastic customer service, Zoho, and congrats on dealing promptly with this attack with honesty and integrity and without patronising anyone. Keep up the good work! Best wishes from remote Scotland.

  3. Hey Zoho - Unfortunately we'll have to pull subscriptions to the Zoho suite unless you get yourselves sorted out very quickly. Response times abysmal on our superfast broadband connection - can't use the product. Very surprised you got caught out like this.

    1. Thanks for your note Hugh. We understand your situation and wish it were different. Under extraordinary situations like a DDoS that has hit multiple vendors in just this week, we cannot make false promises on the restoration of normal service performance. We're sparing no effort, in any quarter, to sort this out for our customers. We appreciate your patience and hope you will stay with us.

  4. Very sorry you've been hit with this ridiculous extortion attempt. No person or company is immune these days, and we appreciate what a terrible position this puts you in. We love your service and we aren't going anywhere. Stay strong!

  5. Wow, what a rush... DDOS attacks aren't easy, ever. Glad you guys could fix it. However I have one thing to report, the custom login URL (business.zoho.com) doesn't seem to be working.

  6. hello Zoho, you guys have to try a way to have everything fixed, i am having serious problem running my business and replying my customers. there is too much delay in receiving and sending email. please you guys have to do your best.

  7. We are unable to receive emails from outside domain. Its a tough time for Zoho and comes out for us as well. Hope it is up and running soon! Well I would like to know that will I get the emails which I did not receive during this time or it will be lost? And yes Shit happens but you guys are putting in great efforts, hope things get well soon.

    1. You should get all your emails. They are just backed up somewhere in the network. Please read the other blog on "Answers to request questions" which we will keep updated.

  8. As per your mail = "Make the following changes in Zoho Sites builder. Login to Zoho Sites, go to Manage >> Settings >> Domains. Make www.yourdomain.com (domain with www) as your primary domain." I am not able to change secondary domain with WWW to primary. It seems to be disabled. Also my sites giving message as "This webpage has a redirect loop". PLEASE LOOK INTO THIS

  9. Reports are not working properly - still. Any business this size should have DDoS protection. It does costs money, but it is worth the price. Please get it fixed. This is NOT a free service. I pay good money so things like this DON'T happen.

    1. John, I understand your frustration. You are running a business and expect your software to just work. We get that. But I call upon you to recognize extraordinary events and avoid improper characterizations. Many large, well-known companies, including most US banks, have been hit by DDoS attacks. Your protection is only as good as the attacker's sophistication. We, like many much larger companies, fell short and are responding fast to fix it.

  10. Well,it's really unstable now,we can not login now,I appreciate the free service zoho has offered, and hope you guys work on this qucik and effective, fight back.This must come from your competitor !

  11. Best of wishes for a speedy recovery. I don't know if this is the proper venue for this, but I wanted to know if anyone else is experiencing issues receiving email from external (non-zoho hosted) sources? I'm not sure how widespread the issue is, but one of my clients that I recommended Zoho to is definitely unable to receive mail from @gmail/@live accounts at this time. Also, at the top of his screen, it reads, "Unable to connect to messaging server". Your fix probably just hasn't rolled out to his organization yet, as my email is working just fine. I will continue to recommend your services, especially due to how fast you folks seem to be able to respond to catastrophic problems such as the one you're dealing with now. Very impressive! THANK YOU FOR YOUR CONTINUED DEDICATION TO MAKING YOUR SERVICES THE BEST ON THE NET!!

  12. We have not been able to access our emails all day today due to these attacks. Being under attack only shows how big the Zoho family is getting! You guys will definitely come out of this far stronger than you came in! Our support remains with Zoho!!!

  13. i am unable to access my website. can some one help me on how to get my website online, the mail i got from Zoho is not clear enough for me to understand the process. the video i wash is from Godaddy domain provider not Zoho. any help please

    1. Ditto rash, Using Network Solutions and unable to make the changes as suggested with them. There is no domain forwarding option. Open to any suggestions.

  14. I still cannot receive any email correspondences, is there anything I can do to have the incoming emails forwarded to an alternative email address? Thank you.

  15. We all come to this point at one time or the other in our cyber-space. Sometimes ago it was SONY, and now it appears its Zoho's turn. We wish Zoho all the best in getting back up. Although too bad, we are in the middle of our Zoho Lagos (pre-conference) for the upcoming event in Jan 2016. We are Zoho Creator Certified Provider in Nigeria. I am afraid to tell my audience that Zoho is currently under some cyber attack but I am proud to tell them that the entire Zoho team is on-top of the game. CHEERS

  16. the issue has started impacting services in India also.... till some time back, though the services were slow in India, now I cant access my emails since last 30 minutes. My support with you and all that are affected in these difficult times.

  17. My website and email are not hosted on Zoho although I am a user of CRM and other services. Can I change anything to have access to CRM during this difficult time?

  18. This is ridiculous, at least the services are back. A lot of our scheduled processes got slammed because of the spotty connection

Leave a Reply

Your email address will not be published.

The comment language code.
By submitting this form, you agree to the processing of personal data according to our Privacy Policy.

Related Posts