Import settings
Use this section to tune the rate and resource utilisation of bulk syncs according to the volume of data you need to sync and the capacity of your system. We have set some default configurations that balance speed and efficiency.
Understanding continuous sync
Bulk contact (Customer, Guest, Subscriber) and insight data syncs (Order, Catalog) run continuously. When the relevant cron job runs, the sync will process all available rows in batches and stop only when everything has been synced.
Example: Syncing 100,000 customers with the default configuration:
ddg_automation_customer_sync starts
Customer IDs fetched from the email_contact table in batches of 500
4 batches of 500 = 2000 - time to create a mega batch
Sync sends data to Dotdigital in 50 mega batches of 2000 each
Tips:
Observe the module's connector.log to see batch counts and sync progress.
It's normal for a single cron job to be in a 'running' state for longer than the configured cron cadence. Equally, it is normal for subsequently scheduled cron jobs to be 'missed' because a previous one is already busy syncing.
Review and Wishlist syncs do not run continuously. They still export a single batch of data, send it to the email_importer table, and wait for the next cron execution. The size of these syncs' data batch is still dictated by the Transactional data batch size.
Batch size vs mega batch size
Continuous sync means a continual cycle of:
query database > export data (get a single ‘batch’)
query database > export data
query database > export data
…
reach the mega batch size limit
send a mega batch to Dotdigital
store the mega batch as JSON in the email_importer table
repeat
A mega batch is, by default, the result of multiple database queries.
Transactional data break
Setting a break value is a way to limit the work done in any single cron 'sync' execution. By default, it is not set to expedite fast data transfer. If you are concerned about resource utilisation, use the break value to sync data in manageable chunks.
If the break value is set, continuous sync will stop when the total number of processed rows exceeds this. The sync then finishes and waits for the next cron run to process more rows. Setting break value applies to all continuous syncs; it effectively makes them non-continuous.
Example: Syncing 100,000 customers with break value set to 20,000:
Cron process begins
5 mega batches are sent to Dotdigital
Sync stops and waits for the next cron execution
Common issues with cron processes and database queries
Overloaded server due to long-running cron processes
Set the Transactional data break to address this issue. Although the sync rate will slow down, it allows for monitoring without the concern of runaway processes.
Impact of Excessive Database Queries on the System
To mitigate this, reduce the Contact sync batch size to limit the number of customers being exported data for in any one loop. Unmap the most query-hungry data fields in Dotdigital > Data Mapping, starting with:
MOST_PUR_CATEGORY
MOST_PUR_BRAND
FIRST_BRAND_PUR
LAST_BRAND_PUR
Cron Tasks Stuck in running for an extended time:
This is expected behavior, as syncs run continuously by default until all eligible data is processed.
Dotdigital sync crons showing as 'missed'
This is also expected behavior. A cron process may over-run its scheduled window if it is processing large amounts of un-processed or not-imported data. Magento's cron scheduler will only spawn one running job of each type at a time and will mark subsequent jobs as 'Missed' until the running job is finished.
Imports failing due to unserialized errors or bad JSON data
This issue likely occurs when the sync attempts to save too much data into the import_data column fails. To resolve this, adjust the relevant mega batch size. This problem should occur less frequently after v4.27.0.
Reducing single batch size configs to low numbers can be counter-productive since it will result in a proportionally larger number of database queries.
Dotdigital will accept up to 50MB of JSON in a single import request. This is a lot of JSON, and trying to maximise mega batch size to meet this limit is not advised. Depending on the number of data fields you have chosen to sync, or the number of order and product attributes you have selected, 50MB might equate to 100,000-150,000 rows. Moving this much data around, serializing and unserializing it, reading and writing it, will be slow and may lead to timeouts.
Tips
Reducing single batch size configs to low numbers can be counter-productive, since it will result in a proportionally larger number of database queries.
Dotdigital will accept up to 50MB of JSON in a single import request. This is a lot of JSON, and trying to maximise mega batch size to meet this limit is not advised. Depending on the number of data fields you have chosen to sync, or the number of order and product attributes you have selected, 50MB might equate to 100,000-150,000 rows. Moving this much data around, serializing and unserializing it, reading and writing it, will be slow and may lead to timeouts.
Sync settings
Automap data fields
If you do not want to manually map the key information that you store on customers from your website to Dotdigital contact data fields, then you can use this option to do it automatically.
To automatically map your data fields:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Admin Task.
For Automap Datafields, select Run Now. All relevant contact data fields are immediately created in your Dotdigital account to match your customer, address, and sales information fields.
Delete all contact IDs
If you want to synchronise your store with a different Dotdigital account, or even synchronise a different store front to a new Dotdigital account. To ensure this works correctly you need to delete all the customer contact IDs that have been stored against the customers, as these are unique to the Dotdigital account.
To delete all contact IDs:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Admin Task.
For Delete All Contact Ids, select Run Now. All contact IDs are immediately deleted from your account.
Manually run sync
Instead of waiting for the next automatic contact data synchronisation in your account, you can manually run one instead.
To run a manual sync:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Manually Run Sync.
For each contact data you want run a sync for, select Run Now. Synchronisation immediately starts.
Customer, Subscriber, Order, Catalog, Wishlist syncs create a batch, but do not import it. You need to select Importer sync to import the batch after creating it.
Setting | Description |
Customer Sync | Syncs your customers to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Subscriber Sync | Syncs your subscribers to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
SMS Subscriber Sync | Syncs your SMS subscribers to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Guest Sync | Syncs your guests to your mapped Dotdigital list. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Order Sync | Syncs your order data to your Dotdigital account. |
Wishlist Sync | Syncs your wishlist data to your Dotdigital account. |
Review Sync | Syncs your review data to your Dotdigital account. |
Catalog Sync | Syncs your catalog data to your Dotdigital account. |
Importer Sync | Syncs your importer data to your Dotdigital account. |
Template Sync | Syncs your template data to your Dotdigital account. |
Reset Sync
You can refresh all of the customer data stored in Dotdigital that you've already imported from your store, marking it all for import again and ensuring it's up to date.
To reset your customer data:
Go to Stores > Configuration > DOTDIGITAL > Developer and find the heading Reset Sync Options.
For Date Range, select the From and To dates you want to reset your customer data from.
If no date range is set then a full data refresh runs.
For each contact data type you want reset, select Run Now. The reset immediately starts.
Setting | Description |
Reset Contacts | Resets the contact data in your account from the data range. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Reset Subscribers | Resets the subscriber data in your account from the data range. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Reset SMS Subscribers | Resets the SMS subscriber data in your account from the data range. Learn more in Map lists in Magento Open Source and Adobe Commerce. |
Reset Order Data | Resets the order data in your account from the data range. |
Reset Review Data | Resets the review data in your account from the data range. |
Reset Wishlist Data | Resets the wishlist data in your account from the data range. |
Reset Catalog Data | Resets the catalog data in your account from the data range. |
Reset B2B Quotes | Resets the B2B quotes in your account from the data range. |
Enable Subscribers Sales Data Sync
This setting can be enabled to sync subscribers’ sales data to Dotdigital.
Enabling this setting increases server load and overall sync times.
Expand the Enable Subscribers Sales Data Sync drop-down menu and select Yes.
Select Save Config.
Enable Remove /pub directory from URLs
This setting can be enabled to remove the /pub directory from image URLs in catalog sync and insight data:
Expand the Remove /pub directory from URLs drop-down menu and select Yes.
Select Save Config.
Debug
Enable debug mode on your website
Expand the Debug Mode drop-down menu and select Enable.
For Debug Api Calls Time, enter a number of seconds.
Any api calls that take longer than the number entered to run are logged.
Select Save Config.
OAUTH Credentials
Find the credentials you want to update and enter valid credentials.
Select Save Config.
Setting | Description |
Client ID | Enter your Client ID. |
Client Secret Key | Enter your Client secret key. |
Connection | Select Connect to open your connected Dotdigital account. |
Custom OAUTH Domain | Enter the custom domain for authorization.
|
Custom Authorization URL | Enter the custom authorization url.
|
Dynamic Pages IP Restriction
When you install the Dotdigital for Adobe Commerce connector, we generate a unique passcode that is used to access external dynamic content. By using this passcode you can be sure that basket contents and coupon codes can be viewed only if that passcode is sent with the GET request to the external dynamic content URL, adding security.
Learn more in Edit your dynamic content security settings in Magento Open Source and Adobe Commerce.
You can also enter a list of requesting IP addresses that these URLs will display for when requested from them:
For Dynamic Pages IP Restriction, enter an IP address.
If you want to enter multiple IP addresses, make sure to separate them with a comma and a space. For example, 52.174.92.164, 20.54.90.19.
Make sure you keep the latest dotdigital IP ranges included. Learn more in IP addresses used by Dotdigital.
Select Save Config.
Manage Cron Timings
Set how frequently you want cron jobs to run:
Expand the drop-down menu for the cron timing you want to edit and choose a time interval.
Select Save Config.
Setting | Description | Options |
Importer | Set how frequently you want the importer cron to run. |
|
Customer | Set how frequently you want the customer cron to run. |
|
Guest | Set how frequently you want the guest cron to run. |
|
Subscriber | Set how frequently you want the subscriber cron to run. |
|
Order | Set how frequently you want the order cron to run. |
|
Catalog | Set how frequently you want the catalog cron to run. |
|
Review And Wishlist | Set how frequently you want the review and wishlist cron to run. |
|
Consent | Set how frequently you want the consent cron to run. |
|
Cleaner | Set how frequently you want the cleaner cron to run. |
|
System alerts
System alerts can notify you when certain functions go wrong in your account.
Enable system messages
When enabled, this setting allows you to send in-app system messages to admin users when errors that may affect the Dotdigital extension are detected:
Expand the Enable System Messages drop-down menu and select Yes.
Select Save Config.
Enable email notifications
When enabled, this setting allows you to send system alert email notifications when errors that may affect the Dotdigital extension are detected:
Expand the Enable Email Notifications drop-down menu and select Yes.
For Select Roles, choose which users you want email notifications to be sent to.
You can select multiple roles by holding the Ctrl key (Windows) or Cmd key (Mac) while selecting.
Expand the Alert Frequency drop-down menu and choose a time period.
This setting controls both email notification frequency and the time period in which the error is reported. For example, select 24 Hours to receive an email every 24 hours, reporting any errors that may have occurred in the last 24 hours.
Select Save Config.
PWA settings
Use this setting to specify if you use a PWA storefront URL that is different from your account’s default frontend base URL:
For PWA Storefront Base URL, enter the URL of your PWA storefront.
For example, https://pwa.yourdomain.com/
Select Save Config.