Understanding Read Operations: Delta vs. Backfill

Integration Fox performs two types of read operations to synchronise data between platforms: Delta and Backfill. These operations ensure your integrations remain efficient and reliable, even when handling large datasets. Below, we explain how each works, their use cases, and the factors that influence their performance.

1. Delta Read: Incremental Synchronisation

What It Does

A Delta Read retrieves only the records created or modified since the last successful sync. This lightweight operation is designed for frequent, near-real-time updates.

When It’s Used

• Default mode for ongoing integrations.

• Ideal for routine updates (e.g., daily CRM changes, new e-commerce orders).

Advantages

• Speed: Processes only recent changes, completing in seconds or minutes.

• Low Resource Usage: Minimises API calls and system strain.

Example Scenario

If 50 contacts were updated in your CRM, a Delta Read fetches only those 50 records and ignores unchanged data.

Limitations

• Depends on the source platform’s ability to track changes.

• If the platform cannot provide incremental data (e.g., due to API limitations), Integration Fox falls back to a Backfill.

2. Backfill Read: Full Data Synchronisation

What It Does

A Backfill Read retrieves all records from the specified entity (e.g., contacts, products). This operation ensures a complete dataset is synced but requires significantly more time and resources.

When It’s Used

• Initial Integration Setup: To establish a baseline dataset.

• Large-System Changes: After bulk imports, migrations, or schema updates.

Factors Affecting Duration

• API Speed: Slower APIs (e.g., rate-limited or paginated endpoints) prolong the process.

• Record Volume: Syncing 100,000 records takes longer than 1,000.

• Data Complexity: Objects with custom fields, associations, or nested data require additional processing.

Limitations of Backfill Reads

• No Syncing During Backfill: Integration Fox pauses all synchronisation for the affected entity while a Backfill is running. This prevents duplicates or conflicting updates, as processing changes mid-backfill could compromise data integrity.

• Resume Normal Operations Post-Backfill: Once the Backfill completes, Delta Reads or incremental syncs resume automatically.

Example Scenario

If a marketing team imports 10,000 new contacts via a CSV upload, the CRM may require a Backfill. During this time, updates to record types are paused until the Backfill finishes.

Key Differences at a Glance

Why Backfills Are Sometimes Necessary

While Delta Reads are preferred, Backfills ensure data accuracy in critical scenarios:

• Data Integrity: Guarantees no records are missed during initial syncs or after major system changes.

• Platform Limitations: Some APIs lack incremental sync capabilities, forcing a full re-sync.