Create Bulk Sync | Polytomic Docs

Create a new Bulk Sync from a source to a destination (data warehouse, database, or cloud storage bucket like S3).

Bulk Syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3.

All of the functionality described in the product documentation is configurable via the API.

Sample code examples:

Connection specific configuration

The destination_configuration is integration-specific configuration for the selected bulk sync destination. This includes settings such as the output schema and is required when creating a new sync.

The source_configuration is optional. It allows configuration for how Polytomic reads data from the source connection. This will not be available for integrations that do not support additional configuration.

Consult the connection configurations to see configurations for particular integrations (for example, here is the available source configuration for the PostgreSQL bulk sync source).

Create a new Bulk Sync from a source to a destination (data warehouse, database, or cloud storage bucket like S3). Bulk Syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3. All of the functionality described in [the product documentation](https://docs.polytomic.com/docs/bulk-syncs) is configurable via the API. Sample code examples: - [Bulk sync (ELT) from Salesforce to S3](https://apidocs.polytomic.com/guides/code-examples/bulk-sync-elt-from-salesforce-to-s-3) - [Bulk sync (ELT) from Salesforce to Snowflake](https://apidocs.polytomic.com/guides/code-examples/bulk-sync-elt-from-salesforce-to-snowflake) - [Bulk sync (ELT) from HubSpot to PostgreSQL](https://apidocs.polytomic.com/guides/code-examples/bulk-sync-elt-from-hub-spot-to-postgre-sql) ## Connection specific configuration The `destination_configuration` is integration-specific configuration for the selected bulk sync destination. This includes settings such as the output schema and is required when creating a new sync. The `source_configuration` is optional. It allows configuration for how Polytomic reads data from the source connection. This will not be available for integrations that do not support additional configuration. Consult the [connection configurations](https://apidocs.polytomic.com/2024-02-08/guides/configuring-your-connections/overview) to see configurations for particular integrations (for example, [here](https://apidocs.polytomic.com/2024-02-08/guides/configuring-your-connections/connections/postgre-sql#source-1) is the available source configuration for the PostgreSQL bulk sync source).

Authentication

AuthorizationBearer

Bearer authentication of the form Bearer <token>, where token is your auth token.

Request

This endpoint expects an object.

destination_configurationmap from strings to anyRequired

destination_connection_idstringRequiredformat: "uuid"

namestringRequired

scheduleobjectRequired

source_connection_idstringRequiredformat: "uuid"

activebooleanOptional

automatically_add_new_fieldsenumOptional

Allowed values:

automatically_add_new_objectsenumOptional

Allowed values:

concurrency_limitintegerOptional

Override the default concurrency limit for this sync.

data_cutoff_timestampdatetimeOptional

disable_record_timestampsbooleanOptionalDefaults to false

discoverbooleanOptional

DEPRECATED: Use automatically_add_new_objects/automatically_add_new_fields instead

modeenumOptional

Allowed values:

normalize_namesenumOptional

Allowed values:

organization_idstringOptionalformat: "uuid"

policieslist of stringsOptional

resync_concurrency_limitintegerOptional

Override the default resync concurrency limit for this sync.

schemaslist of strings or objectsOptional

List of schemas to sync; if omitted, all schemas will be selected for syncing.

source_configurationmap from strings to anyOptional

Response

dataobject

Errors

400

Bulk Sync Create Request Bad Request Error

401

Bulk Sync Create Request Unauthorized Error

403

Bulk Sync Create Request Forbidden Error

422

Bulk Sync Create Request Unprocessable Entity Error

500

Bulk Sync Create Request Internal Server Error

Create a new Bulk Sync from a source to a destination (data warehouse, database, or cloud storage bucket like S3).

Bulk Syncs are used for the ELT pattern (Extract, Load, and Transform), where you want to sync un-transformed data to your data warehouses, databases, or cloud storage buckets like S3.

All of the functionality described in the product documentation is configurable via the API.

Sample code examples:

Connection specific configuration

Consult the connection configurations to see configurations for particular integrations (for example, here is the available source configuration for the PostgreSQL bulk sync source).

1	from polytomic import BulkSchedule, Polytomic
2
3	client = Polytomic(
4	version="YOUR_VERSION",
5	token="YOUR_TOKEN",
6	)
7	client.bulk_sync.create(
8	destination_configuration={"schema": "my_schema"},
9	destination_connection_id="248df4b7-aa70-47b8-a036-33ac447e668d",
10	name="My Bulk Sync",
11	schedule=BulkSchedule(
12	frequency="manual",
13	),
14	source_connection_id="248df4b7-aa70-47b8-a036-33ac447e668d",
15	)

1	{
2	"data": {
3	"active": true,
4	"automatically_add_new_fields": "all",
5	"automatically_add_new_objects": "all",
6	"concurrency_limit": 50,
7	"created_at": "2024-01-15T09:30:00Z",
8	"created_by": {
9	"id": "12345678-1234-1234-1234-123456789012",
10	"name": "John Doe",
11	"type": "user"
12	},
13	"data_cutoff_timestamp": "2023-04-25T12:00:00Z",
14	"destination_configuration": {
15	"schema": "my_schema"
16	},
17	"destination_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
18	"disable_record_timestamps": false,
19	"id": "248df4b7-aa70-47b8-a036-33ac447e668d",
20	"mode": "snapshot",
21	"name": "My Bulk Sync",
22	"normalize_names": "enabled",
23	"organization_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
24	"policies": [
25	"248df4b7-aa70-47b8-a036-33ac447e668d"
26	],
27	"resync_concurrency_limit": 10,
28	"schedule": {
29	"frequency": "manual",
30	"day_of_month": "1",
31	"day_of_week": "monday",
32	"hour": "0",
33	"minute": "0",
34	"month": "1"
35	},
36	"source_configuration": {
37	"replication_slot": "slot"
38	},
39	"source_connection_id": "248df4b7-aa70-47b8-a036-33ac447e668d",
40	"updated_at": "2024-01-15T09:30:00Z",
41	"updated_by": {
42	"id": "12345678-1234-1234-1234-123456789012",
43	"name": "John Doe",
44	"type": "user"
45	},
46	"discover": true
47	}
48	}

Connection specific configuration

Authentication

Headers

Request

Response

Errors

Connection specific configuration