Skip to main content

Set up a data firehose

Learn to set up a data firehose to your external data storage.

Gareth Burroughes avatar
Written by Gareth Burroughes
Updated today

Syncing your event data with Data firehose makes it easier to combine all of the data from across your systems in one place. This allows you to create a fuller picture and exposes more meaningful insights, helping you to make data-driven decisions.

Data firehose offers three options for sync frequency: every 15 minutes, hourly, or daily.


1. Connect a data firehose

  1. In Dotdigital, go to Connect > Data firehose.

  2. Select NEW DATA FIREHOSE, then choose your data transfer method.

    Options:

    • SFTP/FTPS

    • S3: Amazon S3 storage

    • Azure Blob Storage

    • Google Cloud Storage

  3. Enter your SFTP or FTP details into the connect form.

    See the requested form fields:

  4. Select CONNECT.

Amazon S3 form fields

Label

Description

Bucket name

The name of the S3 bucket where the data will be stored.

Region endpoint

The AWS region where the S3 bucket is located.

Folder path

The specific folder path within the S3 bucket where the data will be stored.

Add accountID suffix

A suffix to be added to the account ID for identification purposes.

Access key

The access key for authenticating with AWS.

Secret access key

The secret access key for authenticating with AWS.

SFTPS form fields

Label

Description

FTP type

The type of FTP connection.

Host

The hostname or IP address of the SFTP server.

Port

The port number for the SFTP connection.

Folder path

The specific folder path on the SFTP server. Example: /

Add accountID suffix

A suffix to be added to the account ID for identification purposes.

Connect with

The method of authentication for the SFTP connection. Options: Login or Private key

Username

The username for authenticating with the SFTP server.

Password

The password for authenticating with the SFTP server.

Private key

The private key for authenticating with the SFTP server.

FTPS form fields

Label

Description

FTP type

The type of FTP connection. Example: FTPS

Host

The hostname or IP address of the FTPS server.

Port

The port number for the FTPS connection.

Folder path

The specific folder path on the FTPS server.

Add accountID suffix

A suffix to be added to the account ID for identification purposes.

Username

The username for authenticating with the FTPS server.

Password

The password for authenticating with the FTPS server.

Azure Blob Storage form fields

Label

Description

Account name

To identify the account connection.

Container name

Name of the Azure Blob Storage container to export to.

Folder path

Folder path within the connected container.

Select the Add accountID suffix check box to append your Dotdigital account ID to the folder path name.

This makes it easier to identify specific sync tasks.

  1. For Connect with select from:

  2. Select CONNECT.

Google Cloud Storage form fields

Label

Description

Bucket name

Name of the storage bucket to export to.

Folder path

Folder path within the connected bucket.

Select the Add accountID suffix check box to append your Dotdigital account ID to the folder path name.

This makes it easier to identify specific sync tasks.

Service account key

To authenticate the connection.

Learn how to create a Service account key in Google’s documentation.


2. Choose events to firehose

Next, you must choose the data for the events you want to sync. There are two options here: Engagement or Campaign.

  1. To sync event data, select the checkbox next to the event you want to sync.

  2. Select NEXT.

Event types

Learn about all available event types in Dotdigital Data firehose event schema.


3. Choose a frequency

At this step, you choose how often you want to sync your data.

  1. Expand the Frequency drop-down menu, then choose from:

    • Every 15 minutes

    • Hourly

    • Daily

  2. Expand the Sync failure notification drop-down menu and choose how you would like to be notified if a sync fails:

    • None

    • Email

    • In-app notification

    • Email and in-app notification

    If you select an email notification, enter the email address to notify in the Notification email address field.

  3. Select NEXT.


4. Confirm your firehose setup

Finally, review and confirm your firehose setup.

  1. Check the confirmation details displayed on this page.

  2. Select SYNC.

Did this answer your question?