Duplicate contacts are created: different contacts with the same email address

[IonutOjicaDE]

I just tested the actual version of the plugin and it works! Exactly as described. I am on debian 12, php 8.0, mautic 4.4.11 and MariaDB10.

The only issue I have and I do not know how to solve it:

• the contact confirmed his email address from a previous form registration
• this contact subscribes to a new material, a new form:
• before doi, there is a new contact created with the email address saved on the custom field.
• after doi, the new contact remains and the email address from the custom field is saved into the email field.

At the last step should be a checking if the contact exists already and that contact should be used further on.
Or am I missing something?

By the way : thank you for the plugin !

[sebastian-fahrenkrog]

HI Ionut

Thank you for your feedback. Yes, I'm aware of this problem. I think to resolve this issue, I have to trigger the lead merge action from mautic. Thats how Mautic does it.

Mautic triggers this merging normally if:

1. you call the console command to merge leads
2. if you submit a form with a mapped email lead field
3. if you manually merge leads in the backend

So we need to add this code that handles the lead merging (normally happening after the form submit):
https://github.com/mautic/mautic/blob/d1a8dcdf235f5b50bd1e05146a704e0c250c25da/app/bundles/FormBundle/Model/SubmissionModel.php#L1053

To this situation in the code here:

mautic-doi-plugin/src/JotaworksDoiBundle/Helper/DoiActionHelper.php

Line 147 in 0782088

LeadHelper::leadFieldUpdate($leadFieldUpdate, $this->leadModel, $lead, $ip );

So after we update the lead we should trigger the lead merging (just in case we fill the email lead field for example).
This should then merge the existing lead and the new lead.

Thats the plan and it's on my list but at the moment i'm pretty busy.

Greetings
Sebastian

[IonutOjicaDE]

Thank you for the direction. I succeeded to integrate this functionality. I cannot (or I do not know how) create a pull request, to make the changes directly on the files, but I added the changes below.

Have a great day Sebastian!
Greetings from Schwäbisch Hall, Germany :)

current config.php, from lines 46 to 50:

        'helpers' => [
            'jw.doi.actionhelper' => [
                'class' => \MauticPlugin\JotaworksDoiBundle\Helper\DoiActionHelper::class,
                'arguments' => ['event_dispatcher', 'mautic.helper.ip_lookup', 'mautic.page.model.page', 'mautic.email.model.email', 'mautic.core.model.auditlog', 'mautic.lead.model.lead', 'request_stack' ]
            ],

new config.php starting with line 46:

        'helpers' => [
            'jw.doi.actionhelper' => [
                'class' => \MauticPlugin\JotaworksDoiBundle\Helper\DoiActionHelper::class,
                'arguments' => [
                  'event_dispatcher',
                  'mautic.helper.ip_lookup',
                  'mautic.page.model.page',
                  'mautic.email.model.email',
                  'mautic.core.model.auditlog',
                  'mautic.lead.model.lead',
                  'mautic.lead.model.field',
                  'doctrine.orm.entity_manager',
                  'monolog.logger.mautic',
                  'request_stack' ]
            ],

current DoiActionHelper.php:

<?php

namespace MauticPlugin\JotaworksDoiBundle\Helper;

use Mautic\LeadBundle\Event\ContactIdentificationEvent;
use Mautic\LeadBundle\LeadEvents;
use MauticPlugin\JotaworksDoiBundle\Event\DoiSuccessful;
use MauticPlugin\JotaworksDoiBundle\Helper\LeadHelper;
use MauticPlugin\JotaworksDoiBundle\DoiEvents;

class DoiActionHelper {

    protected $eventDispatcher;

    protected $ipLookupHelper;

    protected $pageModel;

    protected $emailModel;

    protected $auditLogModel;

    protected $leadModel;

    protected $request; 

    public function __construct($eventDispatcher, $ipLookupHelper, $pageModel, $emailModel, $auditLogModel, $leadModel, $request ) 
    {
        $this->eventDispatcher = $eventDispatcher;
        $this->ipLookupHelper = $ipLookupHelper;
        $this->pageModel = $pageModel;
        $this->emailModel = $emailModel;
        $this->auditLogModel = $auditLogModel;
        $this->leadModel = $leadModel;
        $this->request = $request->getCurrentRequest();
    }

... ... ...

    public function updateLead($config) {

        $addTags    = (!empty($config['add_tags'])) ? $config['add_tags'] : [];
        $removeTags = (!empty($config['remove_tags'])) ? $config['remove_tags'] : [];            
        $addTo      = (!empty($config['addToLists'])) ? $config['addToLists']: [];
        $removeFrom = (!empty($config['removeFromLists'])) ? $config['removeFromLists']: [];
        $leadFieldUpdate = (!empty($config['leadFieldUpdate'])) ? $config['leadFieldUpdate']: [];
        $leadFieldUpdateBefore = (!empty($config['leadFieldUpdateBefore'])) ? $config['leadFieldUpdateBefore']: [];

        $lead = $this->leadModel->getEntity($config['lead_id']);
        if(!$lead)
        {
            return;
        }

        // Change Tags (if any)
        if(!empty($addTags)|| !empty($removeTags)){
            $this->leadModel->modifyTags($lead, $addTags, $removeTags);
        }

        // Add to Lists (if any)
        if (!empty($addTo)) {
            $this->leadModel->addToLists($lead, $addTo);
        }

        // Remove from Lists (if any)
        if (!empty($removeFrom)) {
            $this->leadModel->removeFromLists($lead, $removeFrom);
        }       

        //Update lead value (if any)
        if( !empty($leadFieldUpdate) )
        {
            $ip = $this->ipLookupHelper->getIpAddressFromRequest();            
            LeadHelper::leadFieldUpdate($leadFieldUpdate, $this->leadModel, $lead, $ip );               
        }
    }

}

new DoiActionHelper.php:

<?php

namespace MauticPlugin\JotaworksDoiBundle\Helper;

use Mautic\LeadBundle\Event\ContactIdentificationEvent;
use Mautic\LeadBundle\LeadEvents;
use MauticPlugin\JotaworksDoiBundle\Event\DoiSuccessful;
use MauticPlugin\JotaworksDoiBundle\Helper\LeadHelper;
use MauticPlugin\JotaworksDoiBundle\DoiEvents;

use Psr\Log\LoggerInterface;
use Doctrine\ORM\EntityManager;
use Mautic\LeadBundle\Entity\Lead;
use Mautic\LeadBundle\Model\LeadModel;
use Mautic\LeadBundle\Model\FieldModel as LeadFieldModel;
use Mautic\LeadBundle\Deduplicate\ContactMerger;

class DoiActionHelper {

    protected $eventDispatcher;

    protected $ipLookupHelper;

    protected $pageModel;

    protected $emailModel;

    protected $auditLogModel;

    protected $leadModel;

    protected $leadFieldModel;

    protected $em;

    protected $logger;

    protected $request;

    public function __construct($eventDispatcher, $ipLookupHelper, $pageModel, $emailModel, $auditLogModel, $leadModel, $leadFieldModel, $em, $logger, $request )
    {
        $this->eventDispatcher = $eventDispatcher;
        $this->ipLookupHelper  = $ipLookupHelper;
        $this->pageModel       = $pageModel;
        $this->emailModel      = $emailModel;
        $this->auditLogModel   = $auditLogModel;
        $this->leadModel       = $leadModel;
        $this->leadFieldModel  = $leadFieldModel;
        $this->em              = $em;
        $this->logger          = $logger;
        $this->request         = $request->getCurrentRequest();
    }

... ... ...

    public function updateLead($config) {

        $addTags               = (!empty($config['add_tags'])) ? $config['add_tags'] : [];
        $removeTags            = (!empty($config['remove_tags'])) ? $config['remove_tags'] : [];
        $addTo                 = (!empty($config['addToLists'])) ? $config['addToLists']: [];
        $removeFrom            = (!empty($config['removeFromLists'])) ? $config['removeFromLists']: [];
        $leadFieldUpdate       = (!empty($config['leadFieldUpdate'])) ? $config['leadFieldUpdate']: [];
        $leadFieldUpdateBefore = (!empty($config['leadFieldUpdateBefore'])) ? $config['leadFieldUpdateBefore']: [];

        $lead = $this->leadModel->getEntity($config['lead_id']);
        if(!$lead)
        {
            return;
        }
    
        // Change Tags (if any)
        if(!empty($addTags)|| !empty($removeTags)){
            $this->leadModel->modifyTags($lead, $addTags, $removeTags);
        }

        // Add to Lists (if any)
        if (!empty($addTo)) {
            $this->leadModel->addToLists($lead, $addTo);
        }

        // Remove from Lists (if any)
        if (!empty($removeFrom)) {
            $this->leadModel->removeFromLists($lead, $removeFrom);
        }       

        //Update lead value (if any)
        if( !empty($leadFieldUpdate) )
        {
            $ip = $this->ipLookupHelper->getIpAddressFromRequest();            
            LeadHelper::leadFieldUpdate($leadFieldUpdate, $this->leadModel, $lead, $ip );               
        }

        // Merge duplicate contacts if any
        $this->mergeDuplicates($lead);
    }

    public function mergeDuplicates($lead) {
        //get current lead fields and values
        $leadFields       = $lead->getFields(true);
        
        $uniqueLeadFields = $this->leadFieldModel->getUniqueIdentifierFields();
        // Get profile fields of the current lead
        $currentFields    = $lead->getProfileFields();
    
        // Closure to get data and unique fields
        $getData = function ($currentFields, $uniqueOnly = false) use ($leadFields, $uniqueLeadFields) {
            $uniqueFieldsWithData = $data = [];
            foreach ($leadFields as $alias => $properties) {
                if (isset($currentFields[$alias])) {
                    $value        = $currentFields[$alias];
                    $data[$alias] = $value;

                    // make sure the value is actually there and the field is one of our uniques
                    if (!empty($value) && array_key_exists($alias, $uniqueLeadFields)) {
                        $uniqueFieldsWithData[$alias] = $value;
                    }
                }
            }

            return ($uniqueOnly) ? $uniqueFieldsWithData : [$data, $uniqueFieldsWithData];
        };

        // Closure to help search for a conflict
        $checkForIdentifierConflict = function ($fieldSet1, $fieldSet2) {
            // Find fields in both sets
            $potentialConflicts = array_keys(
                array_intersect_key($fieldSet1, $fieldSet2)
            );

            $this->logger->debug(
                'DOIupdateLead: Potential conflicts '.implode(', ', array_keys($potentialConflicts)).' = '.implode(', ', $potentialConflicts)
            );

            $conflicts = [];
            foreach ($potentialConflicts as $field) {
                if (!empty($fieldSet1[$field]) && !empty($fieldSet2[$field])) {
                    if (strtolower($fieldSet1[$field]) !== strtolower($fieldSet2[$field])) {
                        $conflicts[] = $field;
                    }
                }
            }

            return [count($conflicts), $conflicts];
        };

        // Get data for the form submission
        [$data, $uniqueFieldsWithData] = $getData($currentFields);
        $this->logger->debug('DOIupdateLead: Unique fields submitted include '.implode(', ', $uniqueFieldsWithData));
    
        $existingLeads = (!empty($uniqueFieldsWithData)) ? $this->em->getRepository('MauticLeadBundle:Lead')->getLeadsByUniqueFields(
            $uniqueFieldsWithData,
            $lead->getId()
        ) : [];

        $uniqueFieldsCurrent = $getData($currentFields, true);
        
        if (count($existingLeads)) {
            $this->logger->debug(count($existingLeads).' found based on unique identifiers');

            /** @var \Mautic\LeadBundle\Entity\Lead $foundLead */
            $foundLead = $existingLeads[0];

            $this->logger->debug('Testing contact ID# '.$foundLead->getId().' for conflicts');

            // Get profile fields of the found lead
            $foundLeadFields = $foundLead->getProfileFields();

            $uniqueFieldsCurrent = $getData($currentFields, true);
            $uniqueFieldsFound   = $getData($foundLeadFields, true);
            [$hasConflict, $conflicts] = $checkForIdentifierConflict($uniqueFieldsFound, $uniqueFieldsCurrent);

            if ($hasConflict || !$lead->getId()) {
                $lead = $foundLead;
                if ($hasConflict) {
                    $this->logger->debug('Conflicts found in '.implode(', ', $conflicts).' so not merging');
                }
            } else {
                $this->logger->debug('Merging contacts '.$lead->getId().' and '.$foundLead->getId());
                $lead = $this->leadModel->mergeLeads($lead, $foundLead);
            }
        }
    }
}

[sebastian-fahrenkrog]

HI @IonutOjicaDE
Amazing! Thank you very much! Great Work!

I will do my best to merge it the next days and test it.

Greetings from Wuppertal
Sebastian