Have you ever had the same email sent multiple times to the same customer during a campaign? Ever had the same lead assigned to multiple sales reps, and had to deal with an irate prospect? If you have, then you know that’s a surefire way for losing existing and even potential customers. One of the primary culprits for mistakes like those is duplicate data: records with the same email or same phone numbers, but stored under different names.

 Let’s look at some of the ways that duplicate dataand bad data in generalcan affect your organization.

  • Higher consumption of resources
  • Higher maintenance cost
  • Higher spam counts
  • Higher churn rates
  • Lower customer satisfaction and retention
  • Invalid reports
  • Lowered productivity
  • Loss of revenue

Research by Experian Information solutions shows that businesses are losing 14% of their revenue due to bad data. This equates to a staggering $258.278 million just in the UK alone. But revenue is not the only aspect of a business to take a hit due to rogue data: The research also showed that around 21% of UK businesses have taken a reputation hit as well.

If you wish to conduct your business smoothly without unwanted expenses, get rid of all duplicates! They are detrimental to your growth.

An organization spends on average about $14 million on data quality. Kissmetrics breaks down how much bad data, which includes duplicates, costs the company in terms of revenue, and the difference it makes to have good quality data to work with.

It‘s unreasonable for organizations to spend so much money on data quality. That’s why we’ve made it our mission to ensure that Zoho CRM is capable of inherently assisting an organization in maintaining data quality. 

We’ve approached duplicates like a diseaseFor any affliction, there are two ways you can deal with it. You can either prevent it before it affects you or, in case you’ve already been affected, the only thing you can do is remedy it. It’s the same with duplicate data. 

 The prevention

We’ve introduced something called Unique Fields. You can choose to mark particular fields as unique (ex: name, phone number, etc.) through the layout setting option. Every time a new record with the same variable in any of the fields marked unique is being pushed into the CRM, a prompt pops up saying that a duplicate is about to be created and the action will not be allowed. This is a preventive measure in Zoho CRM to make sure no duplicates are generated.

 The remedy

To deal with duplicates already present in the CRM, we have the de-duplicate option in the Leads, Contacts, Vendors and Accounts module. The de-duplication function is present under the “more options” button. The process starts once you decided upon the fields you wish to check for duplicates.

An automated deduplication process will run in the background, merging records with completely identical variables in the selected fields. Once it’s done, you’ll be notified that the process is complete.

Deduplication in Zoho CRM used to be a manual process, but now we’ve made sure you don’t have to waste time sorting through all those records.  So now, the only manual deduplication you need to worry about is when there are conflicts across a particular field. A prompt will show up after the process is completed to let you know that duplicates with conflicting records were found.

This is the only manual deduplication you need to do since it’s your call on which records need to be merged.  After you’ve thought about it, select which of the conflicting records you want to merge and click “Merge”.

Duplicate data are a pain to deal with once inside your system. Apart from improving your efficiency, maintaining good data quality can have a direct impact on your revenue. The chart from Ringlead below shows that a strong organization can generate about 66% more revenue than an average organization based on the quality of data alone.

(Source: Ringlead)

Maintain a clean database with Zoho CRM. Get started today! Your feedback on what you think can be improved would be of great assistance to us. Please feel free to leave your thoughts in the comments below.

 

 

  1. Richard Dalton

    Improve your customer service. Your customer service reps take a long time to respond in chat, even after the conversation has started.
    Also, some make mistakes because they aren’t very knowledgeable about the system.
    It is so frustrating to the point that I sometimes consider switching to another CRM.

  2. Peter

    CAn you please create a de-duplicate option for the “Deals” tab. My pipeline is twice the size it should be.

    • Shashank

      Hi, Peter. We’re working hard to provide the deduplication for every module. But could we know how you ascertained that your pipeline looks bloated due to duplicates?

      • Steve O'Rourke

        I would suggest that you just create a checkbox next to the list of opportunities under an account, and create a Merge feature to combine them, allowing the user to select which conflicting values to store on the resulting merged record.

  3. Stacie Bennison

    I like that this will run in the background – but would like to see more variables for deduplication of accounts. One account could have same name and phone number but different locations. This would then merge the accounts but really need to remain separate. It would seem that address should be a critical match factor.

    • Shashank

      Hi, Stacie. To address your concern, it would not auto-merge since the auto-merge only occurs for completely identical records. Records with variables like you have mentioned would throw a “conflicts present” result. Hope that clears things up.

  4. Fred Webster

    Prevention is the first step in managing dupes. Our company does not sell to an individual, rather we sell to a company. A user will search for the name of a company within the Accounts module. All records that have a partial of the “search” word will be displayed. IF that results display could be sorted (on any of the results column headers) users would be able to find an existing record quicker rather than scrolling 10 records at a time to find if it is already in the database. The current results grid has no sorting feature.

    • Shashank

      Hi, Fred. Yes, the global search result page doesn’t have a sort feature right now.

      We’ll look into the feasibility of adding that feature in.

  5. Sonia Lub

    i just tried this tool… And afterwards was told that I merged 439 duplicates. How do i see what 439 duplicates have been merged?

    • Shashank

      Hi Sonia,
      As of now, you can’t see them. We are working on to show the master records ( alone ) on which the mirror duplicates are merged automatically. I will keep posted once it is done.

  6. GJ

    Hi team,

    love the function however, you really need to extend this to custom modules, we are a ZOHO partner and build crm systems for the insurance industry, we use custom modules for clients, claims, renewals, policies, endorsements, etc.
    The duplicate problem is not only restricted to leads and accounts, so an extension to custom modules would be very beneficial.

    kind regards GJ

    • Shashank

      Glad to have your feedback GJ!
      But there are scalability challenges. So, we started off with our basic modules. We will definitely open this option to custom modules once we are ready with our scalability challenges. I will keep you posted on this.

  7. Todd Moritz

    Will de-duplicate work only within the LEAD ..OR… CONTACTS module?
    Or, will de-duplicate work across the two modules?

    • Shashank

      Hey Todd! The de-duplicate work with leads and contacts separately. Not across the modules. However, the duplication check will work for leads and converted leads (contacts).

  8. Lorrie Jo

    We would like a pop up (or notification) if we are duplicating a field, but still have the option to create the duplicate lead. Is there something that would do this for us?

    • Shashank

      Sorry, Lorrie. As of now, there is no way that you a get notification of duplication and get away with that. We don’t provide that option. Either allow duplicates or restrict duplicates.

  9. Janice Waterman

    I agree with Gary…please give the first & last name option! My customers move between companies thus the company and email change. Only the name stays the same.
    Thank you!

  10. Gary Reynolds

    I would like to see the first and last name as an option.

    • Shashank

      Good day Gary. We can’t give the combination of the first and last name. I understand your requirement. Any combination of fields is not possible as of now. But, it is because of the technical constraint. We can’t promise anything on this.

  11. Patty Fajardo

    How do I do this? Is there a step by step process?

    • Shashank

      Hey, Patty. Please have a look at this It details how the feature can be used.

  12. Mark Wedemeyer

    How does the automatic duplication process determine ownership of the record, when the records have different owners? Any dupe files with different ownership, should by default be flagged for manual resolution. The records should dump into a common file which can then be sorted through, to determine proper ownership before merging.

    The sales team will go bonkers if any of their accounts are switched by automation to a different owner..

    • Shashank

      Hi, Mark. As of now, the mirror records are determined by leaving the owner field. Very rarely that you will end up having with “ALL THE FIELD VALUES ALIKE AND THE OWNERS, DIFFERENT. So, we don’t consider the owner name as a differential factor, when all the field values are same. If you have any practical-glitch, please let us know. We will see how we can sort it out.

  13. Rachael

    A very helpful solution would be to be able to give permission to “Merge Duplicates” to specific users/groups. Where an organization has a lot of duplicates, they are not likely going to have their executives or IT personnel process them. It would be someone that takes up less resources for their time, who is still familiar with the accounts.

    • Shashank

      Hey there, Rachael!

      The profiles which have “Find and Merge” permission alone can do the “Deduplication”. You can assign the profile- with this permission enabled- to the users you want and manage.

  14. Marty

    Existing in my system already. I have duplicates that are both leads and contacts. Do you have any way to help me match these for deduplication?

    • Shashank

      Hey, Marty. You can’t de-dupe the leads and contacts together. It is applicable only to one module. When you actually convert a lead and when a contact already exists, then the conversion process will take care of, to merge with the existing contact or duplicate it.

      • Steve O'Rourke

        For the record, it is technically feasible to identify duplicates between Leads and Contacts using a union join between the two tables. Certainly might take a little out-of-the-box thinking here, but you all could do it. We could certainly benefit from that.

  15. laura

    Never mind! I see the option in there now. going to test it.Thanks!

  16. Preston

    “records with completely identical variables”

    This is still a problem. If the company name varies slightly such as inc or co, Dedupe will not catch it. I’d prefer that it give suggested dedupe options along with automatically merging exact matches.

    • Shailesh

      Yes, agree with you Preston. Most of the duplicate entries in the system happen due to slight mismatch of names (Accounts and Contacts) even when a search is done to check for duplicates before creating a new record. Having a suggested dedupe option would be very useful. That’s actually how we search for duplicates…by entering partial names to get a better match.

  17. Laura

    Can you add this to all the tabs? I want to de-duplicate People, Companies, etc.

  18. Nikita Thomas

    It would be better to prevent duplicate from being entered before they are put into the CRM. This new deduplication feature only merges profiles that match exactly. Different users may be putting in different data so there is still a manual process.

    You should also consider deduping on not only the unique fields but a combination of unique fields and other fields the company knows that is more indicative of a duplicate. In our company we use Contacts for married couples or people that live in the same building. It would be more effective if we the mass deduplication merged contacts that had the same first name, last name, phone and address.

    This is something that I have been requesting for more than a year and your helpdesk says it’s impossible. But until this change is made, your new feature still does not help to clean up our data.

    • Shashank

      Good day, Nikita. I can’t agree more. The combination of unique field is not required. Since it is unique you can’t duplicate it. But a provision to mark UNIQUENESS based on the combination of other two or three fields should be supported. But, the reason why we did not provide the combination of fields is purely due to the technical constraint. We are striving hard to cross over technical hurdles. It would take time. We will keep you posted on this.

  19. Steve O'Rourke

    I’ve repeatedly suggested that Zoho create a checkbox on records in search results to enable you to merge duplicates right there and then. But you have to select one, then go and find the duplicate again, eliminating select pre-filled fields and checking your and/or logic, just to get back to where you were when you did your initial search. It’s no wonder why our people don’t clean up duplicates more often.

    I’m certainly curious about this automated de-dupe function, but don’t have a high degree of trust that it will run smoothly and without unintentional error. I’d probably prefer to test it on a copy before running it on our live database.

    • Shashank

      Good day Steve!
      This is a nice suggestion. We’ll look into the possibilities.

  20. Brandon Baisas

    What about duplicate data across Leads and Accounts? For example, “The X Company” is a customer in Accounts module. However, marketing uploaded a new lead list to Leads module that also includes “The X Company” in the list. Then sales calls the lead, not knowing that it is also an Account. How can we prevent this from happening?

    • Val

      We have the same problem, can this be solved by the automated de-dupe function update? i’m curious as well but do not trust that it will run smoothly either…

    • Shashank

      Hey, Brandon! Basically, you would like to prevent the duplicate leads from the already converted leads ( i.e the lead would have qualified and converted as an Account and Contact ). Right now, through API, Webforms, and Manual entry the duplicates will be stopped. However, all the other means of lead source will be covered soon. ( Import, Contact sync, etc., )

      But, the deduplication ( remedy ) across the modules can’t be done.

      • Steve O'Rourke

        Just as with leads & contacts, it is technically feasible to identify duplicates between Leads and Accounts using a union join between the two tables. Again, this will require a little out-of-the-box thinking, but it could be done. This would be a big benefit to a lot of us!

  21. Lorrie Jo

    Can you override the function that does not allow you to create a duplicate if you need one?

    • Shashank

      Good day, Lorrie. You can select or deselect the “Do not allow duplicate values” option.

      Please refer to this forum post and this help documentation
      Cheers,
      Shashank

  22. Julien Grandjean

    This is a great news, and we tested this feature last week. Really amazing. Thanks Zoho.

    • Shashank

      Happy to know you like the feature. Let us know how it helped you and what we can do to make your Zoho CRM experience better!
      Have a great day!

  23. Rehaf

    Yes! I really wanted the unique field option and now that you have it, you’ll make my sales life easier

    • Shashank

      Glad we could help you Rehaf! Do let us know your thoughts on anything that could be added or improved.

      Cheers,
      Shashank.