Import settings
Use this section to tune the rate and resource utilisation of bulk syncs according to the volume of data you need to sync and the capacity of your system. We have set some default configurations that balance speed and efficiency.
Understand continuous sync
Bulk contact (Customer, Guest, Subscriber) and Insight data syncs (Order, Catalog) run continuously. When the relevant cron job runs, the sync processes all available rows in batches and stops only when everything has been synced.
Example: Syncing 100,000 customers with the default configuration:
ddg_automation_customer_sync
startsCustomer IDs fetched from the
email_contact
table in batches of 5004 batches of 500 = 2000 - time to create a mega batch
Sync sends data to Dotdigital in 50 mega batches of 2000 each
Tips:
Observe the module's connector.log to see batch counts and sync progress.
It's normal for a single cron job to be in a running state for longer than the configured cron cadence. Equally, it is normal for subsequently scheduled cron jobs to be missed because a previous one is already busy syncing.
Review and Wishlist syncs do not run continuously. They still export a single batch of data, send it to the
email_importer
table, and wait for the next cron execution. The size of these syncs' data batch is still dictated by the Transactional data batch size.
Batch size vs mega batch size
Continuous sync means a continual cycle of:
query database > export data (get a single ‘batch’)
query database > export data
query database > export data
…
reach the mega batch size limit
send a mega batch to Dotdigital
store the mega batch as JSON in the
email_importer
tablerepeat
A mega batch is, by default, the result of multiple database queries.
Transactional data break
Setting a break value is a way to limit the work done in any single cron sync execution. By default, it is not set to expedite fast data transfer. If you are concerned about resource utilisation, use the break value to sync data in manageable chunks.
If the break value is set, continuous sync stops when the total number of processed rows exceeds this. The sync then finishes and waits for the next cron run to process more rows. Setting break value applies to all continuous syncs; it effectively makes them non-continuous.
Example: Syncing 100,000 customers with break value set to 20,000:
Cron process begins
5 mega batches are sent to Dotdigital
Sync stops and waits for the next cron execution
Common issues with cron processes and database queries
Overloaded server due to long-running cron processes
Set the Transactional data break to address this issue. Although the sync rate slows down, it allows for monitoring without the concern of runaway processes.
Impact of excessive database queries on the system
To mitigate this, reduce the Contact sync batch size to limit the number of customers data is being exported for in any one loop. Un-map the most query-hungry data fields in Dotdigital > Data Mapping, starting with:MOST_PUR_CATEGORY
MOST_PUR_BRAND
FIRST_BRAND_PUR
LAST_BRAND_PUR
Cron Tasks Stuck in running for an extended time
This is expected behaviour, as syncs run continuously by default until all eligible data is processed.
Dotdigital sync crons showing as Missed
This is also expected behaviour. A cron process may over-run its scheduled window if it is processing large amounts of unprocessed or unimported data. Magento's cron scheduler only spawns one running job of each type at a time, and marks subsequent jobs as Missed until the running job is finished.
Imports failing due to unserialized errors or bad JSON data
This issue likely occurs when the sync attempts to save too much data into theimport_data
column fails. To resolve this, adjust the relevant mega batch size. This problem should occur less frequently after v4.27.0.
Tips
Reducing single batch size configs to low numbers can be counter-productive, since it results in a proportionally larger number of database queries.
Dotdigital accepts up to 50MB of JSON in a single import request. This is a lot of JSON, and trying to maximise mega batch size to meet this limit is not advised.
Depending on the number of data fields you have chosen to sync, or the number of order and product attributes you have selected, 50MB might equate to 100,000-150,000 rows. Moving this much data around, serialising and un-serialising it, reading and writing it, is slow and may lead to timeouts.
Sync settings
Automap data fields
If you don't want to manually map the key information that you store on customers from your website to Dotdigital contact data fields, then you can use this option to do it automatically.
To automatically map your data fields:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Admin Task.
For Automap Datafields, select Run Now. All relevant contact data fields are immediately created in your Dotdigital account to match your customer, address, and sales information fields.
Delete all contact IDs
You might want to synchronise your store with a different Dotdigital account, or even synchronise a different store front to a new Dotdigital account. To ensure this works correctly, you need to delete all the customer contact IDs that have been stored against the customers, as these are unique to the Dotdigital account.
To do this:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Admin Task.
For Delete All Contact Ids, select Run Now. All contact IDs are immediately deleted from your account.
Manually run sync
Instead of waiting for the next automatic contact data synchronisation in your account, you can manually run one instead.
To run a manual sync:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Manually Run Sync.
For each contact data you want run a sync for, select Run Now. Synchronisation immediately starts.
Customer, Subscriber, Order, Catalog, Wishlist syncs create a batch, but do not import it. You need to select Importer sync to import the batch after creating it.
Setting | Description |
Customer Sync | Syncs your customers to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Subscriber Sync | Syncs your subscribers to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
SMS Subscriber Sync | Syncs your SMS subscribers to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Guest Sync | Syncs your guests to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Order Sync | Syncs your order data to your Dotdigital account. |
Wishlist Sync | Syncs your wishlist data to your Dotdigital account. |
Review Sync | Syncs your review data to your Dotdigital account. |
Catalog Sync | Syncs your catalog data to your Dotdigital account. |
Importer Sync | Syncs your importer data to your Dotdigital account. |
Template Sync | Syncs your template data to your Dotdigital account. |
Reset Sync
You can refresh all of the customer data stored in Dotdigital that you've already imported from your store, marking it all for import again and ensuring it's up to date.
To reset your customer data:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Reset Sync Options.
For Date Range, select the From and To dates you want to reset your customer data from.
If no date range is set then a full data refresh runs.
For each contact data type you want reset, select Run Now. The reset immediately starts.
Setting | Description |
Reset Contacts | Resets the contact data in your account from the data range. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Reset Subscribers | Resets the subscriber data in your account from the data range. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Reset SMS Subscribers | Resets the SMS subscriber data in your account from the data range. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Reset Order Data | Resets the order data in your account from the data range. |
Reset Review Data | Resets the review data in your account from the data range. |
Reset Wishlist Data | Resets the wishlist data in your account from the data range. |
Reset Catalog Data | Resets the catalog data in your account from the data range. |
Reset B2B Quotes | Resets the B2B quotes in your account from the data range. |
Enable Subscribers Sales Data Sync
This setting can be enabled to sync subscribers’ sales data to Dotdigital.
Enabling this setting increases server load and overall sync times.
Expand the Enable Subscribers Sales Data Sync drop-down menu and select Yes.
Select Save Config.
Enable Remove /pub
directory from URLs
This setting can be enabled to remove the /pub
directory from image URLs in catalog sync and insight data:
Expand the Remove /pub directory from URLs drop-down menu and select Yes.
Select Save Config.
Enable debug mode on your website
Expand the Debug Mode drop-down menu and select Enable.
For Debug Api Calls Time, enter a number of seconds.
Any API calls that take longer than the number entered to run are logged.
Select Save Config.
OAUTH Credentials
Find the credentials you want to update and enter valid credentials.
Select Save Config.
Setting | Description |
Client ID | Enter your Client ID. |
Client Secret Key | Enter your Client Secret Key. |
Connection | Select Connect to open your connected Dotdigital account. |
Custom OAUTH Domain | Enter the custom domain for authorisation.
Make sure that the base URL ends with |
Custom Authorization URL | Enter the custom authorisation URL.
Make sure that the base URL ends with |
Dynamic Pages IP Restriction
When you install the Dotdigital for Adobe Commerce connector, we generate a unique passcode that is used to access external dynamic content. By using this passcode, you can be sure that basket contents and coupon codes can be viewed only if that passcode is sent with the GET request to the external dynamic content URL, adding security.
You can also enter a list of requesting IP addresses that these URLs display for when requested from them:
For Dynamic Pages IP Restriction, enter an IP address.
If you want to enter multiple IP addresses, make sure to separate them with a comma and a space. For example, 52.174.92.164, 20.54.90.19.
Make sure you keep the latest dotdigital IP ranges included.
Learn more in IP addresses used by Dotdigital.
Select Save Config.
Manage Cron Timings
To set how frequently you want cron jobs to run:
Expand the drop-down menu for the cron timing you want to edit and choose a time interval.
Select Save Config.
Setting | Options |
Importer |
|
Customer |
|
Guest |
|
Subscriber |
|
Order |
|
Catalog |
|
Review And Wishlist |
|
Consent |
|
Cleaner |
|
System alerts
System alerts can notify you when certain functions go wrong in your account.
Enable system messages
When enabled, this setting allows you to send in-app system messages to admin users when errors that may affect the Dotdigital extension are detected:
Expand the Enable System Messages drop-down menu and select Yes.
Select Save Config.
Enable email notifications
When enabled, this setting allows you to send system alert email notifications when errors that may affect the Dotdigital extension are detected:
Expand the Enable Email Notifications drop-down menu and select Yes.
For Select Roles, choose which users you want email notifications to be sent to.
You can select multiple roles by holding the Ctrl key (Windows) or Cmd key (Mac) while selecting.Expand the Alert Frequency drop-down menu and select a time period.
This setting controls both email notification frequency and the time period in which the error is reported. For example, select 24 Hours to receive an email every 24 hours, reporting any errors that may have occurred in the last 24 hours.
Select Save Config.
PWA settings
Use this setting to specify if you use a PWA storefront URL that is different from your account’s default frontend base URL:
For PWA Storefront Base URL, enter the URL of your PWA storefront, for example, https://pwa.yourdomain.com/.
Select Save Config.