Create Bulk Sync | Polytomic Docs

curl -X POST https://app.polytomic.com/api/bulk/syncs \
     -H "X-Polytomic-Version: X-Polytomic-Version" \
     -H "Content-Type: application/json" \
     -u "<username>:<password>" \
     -d '{
  "destination_configuration": {
    "schema": "my_schema"
  },
  "destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
  "mode": "snapshot",
  "name": "My Bulk Sync",
  "schedule": {
    "frequency": "manual"
  },
  "source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d"
}'

{
  "data": {
    "active": true,
    "automatically_add_new_fields": "all",
    "automatically_add_new_objects": "all",
    "concurrency_limit": 50,
    "created_at": "2024-01-15T09:30:00Z",
    "created_by": {
      "id": "12345678-1234-1234-1234-123456789012",
      "name": "John Doe",
      "type": "user"
    },
    "data_cutoff_timestamp": "2023-04-25T12:00:00Z",
    "destination_configuration": {
      "schema": "my_schema"
    },
    "destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "disable_record_timestamps": false,
    "id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "mode": "snapshot",
    "name": "My Bulk Sync",
    "normalize_names": "enabled",
    "organization_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "policies": [
      "248df4b7-aa70-47b8-a036-33ac447e668d"
    ],
    "resync_concurrency_limit": 10,
    "schedule": {
      "frequency": "manual",
      "day_of_month": "1",
      "day_of_week": "monday",
      "hour": "0",
      "minute": "0",
      "month": "1",
      "multi": {
        "schedules": [
          {
            "item": "none"
          }
        ],
        "type": "string"
      }
    },
    "source_configuration": {
      "replication_slot": "slot"
    },
    "source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
    "updated_at": "2024-01-15T09:30:00Z",
    "updated_by": {
      "id": "12345678-1234-1234-1234-123456789012",
      "name": "John Doe",
      "type": "user"
    },
    "discover": true
  }
}

Creates a new bulk sync from a source connection to a destination connection.

Bulk syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3.

All of the functionality described in the product documentation is configurable via the API.

Sample code examples:

Connection specific configuration

The destination_configuration is integration-specific configuration for the selected bulk sync destination. This includes settings such as the output schema and is required when creating a new sync.

The source_configuration is optional. It allows configuration for how Polytomic reads data from the source connection. This will not be available for integrations that do not support additional configuration.

Consult the connection configurations to see configurations for particular integrations (for example, here is the available source configuration for the PostgreSQL bulk sync source).

Defaults and selection behavior

If schemas is omitted, the sync is created with all available source schemas selected. Pass schemas explicitly if you want the initial sync to include only a subset of tables or objects.

Schedule times are interpreted in UTC.

When omitted, automatic discovery defaults are conservative:

automatically_add_new_objects defaults to not enabling newly discovered source objects automatically.
automatically_add_new_fields defaults to enabling newly discovered fields on already selected objects.
normalize_names defaults to enabled.

Creates a new bulk sync from a source connection to a destination connection. Bulk syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3. All of the functionality described in [the product documentation](https://docs.polytomic.com/docs/bulk-syncs) is configurable via the API. Sample code examples: - [Bulk sync (ELT) from Salesforce to S3](../../../guides/code-examples/bulk-sync-elt-from-salesforce-to-s-3) - [Bulk sync (ELT) from Salesforce to Snowflake](../../../guides/code-examples/bulk-sync-elt-from-salesforce-to-snowflake) - [Bulk sync (ELT) from HubSpot to PostgreSQL](../../../guides/code-examples/bulk-sync-elt-from-hub-spot-to-postgre-sql) ## Connection specific configuration The `destination_configuration` is integration-specific configuration for the selected bulk sync destination. This includes settings such as the output schema and is required when creating a new sync. The `source_configuration` is optional. It allows configuration for how Polytomic reads data from the source connection. This will not be available for integrations that do not support additional configuration. Consult the [connection configurations](../../../guides/configuring-your-connections/overview) to see configurations for particular integrations (for example, [here](../../../guides/configuring-your-connections/connections/postgre-sql#source-1) is the available source configuration for the PostgreSQL bulk sync source). ## Defaults and selection behavior If `schemas` is omitted, the sync is created with all available source schemas selected. Pass `schemas` explicitly if you want the initial sync to include only a subset of tables or objects. Schedule times are interpreted in UTC. When omitted, automatic discovery defaults are conservative: - `automatically_add_new_objects` defaults to not enabling newly discovered source objects automatically. - `automatically_add_new_fields` defaults to enabling newly discovered fields on already selected objects. - `normalize_names` defaults to enabled.

Authentication

AuthorizationBasic

This is the Polytomic deployment key (on-premises only)

AuthorizationBearer

This is the Polytomic API key

Request

This endpoint expects an object.

destination_configurationmap from strings to anyRequired

destination_connection_idstringRequiredformat: "uuid"

modeenumRequired

namestringRequired

scheduleobjectRequired

source_connection_idstringRequiredformat: "uuid"

activeboolean or nullOptional

automatically_add_new_fieldsenumOptional

automatically_add_new_objectsenumOptional

concurrency_limitinteger or nullOptional

Override the default concurrency limit for this sync.

data_cutoff_timestampdatetime or nullOptional

disable_record_timestampsboolean or nullOptionalDefaults to false

discoverboolean or nullOptional

DEPRECATED: Use automatically_add_new_objects/automatically_add_new_fields instead

normalize_namesenumOptional

organization_idstring or nullOptionalformat: "uuid"

policieslist of strings or nullOptional

resync_concurrency_limitinteger or nullOptional

Override the default resync concurrency limit for this sync.

schemaslist of strings or objects or nullOptional

List of schemas to sync; if omitted, all schemas will be selected for syncing.

source_configurationmap from strings to any or nullOptional

Response

dataobjectOptional

Errors

400

Bad Request Error

401

Unauthorized Error

403

Forbidden Error

422

Unprocessable Entity Error

500

Internal Server Error

Creates a new bulk sync from a source connection to a destination connection.

Bulk syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3.

All of the functionality described in the product documentation is configurable via the API.

Sample code examples:

Connection specific configuration

Consult the connection configurations to see configurations for particular integrations (for example, here is the available source configuration for the PostgreSQL bulk sync source).

Defaults and selection behavior

If schemas is omitted, the sync is created with all available source schemas selected. Pass schemas explicitly if you want the initial sync to include only a subset of tables or objects.

Schedule times are interpreted in UTC.

When omitted, automatic discovery defaults are conservative:

automatically_add_new_objects defaults to not enabling newly discovered source objects automatically.
automatically_add_new_fields defaults to enabling newly discovered fields on already selected objects.
normalize_names defaults to enabled.

Connection specific configuration

Defaults and selection behavior

Authentication

Headers

Request

Response

Errors

Connection specific configuration

Defaults and selection behavior