Bulk sync (ELT) from Salesforce to S3

UI demo video

Watch the same steps in the Polytomic admin console in this demo video.

Code

Set your Polytomic API key as an environment variable:

$ export POLYTOMIC_API_KEY=YOUR-API-TOKEN

This example covers three steps:

Create a Salesforce Connection.
Create an S3 Connection.
Bulk Sync data from Salesforce into S3.

1. Create a Salesforce connection

The following request creates a Salesforce Connection. See the Salesforce connection configuration for the required fields.

cURL

Python

Typescript

Go

$ curl --request POST  \
>      --url https://app.polytomic.com/api/connections \
>      --header "accept: application/json"  \
>      --header "content-type: application/json"  \
>      --header "X-Polytomic-Version: 2024-02-08"   \
>      --header "Authorization: Bearer ${POLYTOMIC_API_KEY}"  \
>      -d '{"name": "Salesforce Connection","type": "salesforce", "configuration": {"domain": "https://example.my.salesforce.com"}}'

Salesforce Connections authenticate with OAuth. Open the URL returned in the auth_url field of the response to complete the flow.

OAuth redirection

By default, the API expects auth_url to open in a new browser window. Set the optional redirect_url parameter in the request body to change the redirect target.

2. Create an S3 connection

The S3 bucket must be reachable from Polytomic over the network. For details, see the S3 connection guide.

cURL

Python

Typescript

Go

$      curl --request POST \
>      --url https://app.polytomic.com/api/connections \
>      --header "accept: application/json" \
>      --header "content-type: application/json" \
>      --header "X-Polytomic-Version: 2024-02-08"  \
>      --header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
>      -d '{
>      "name": "S3 Connection",
>      "type": "s3",
>      "configuration": {
>         "aws_access_key_id": "EXAMPLE",
>         "aws_secret_access_key": "SECRET",
>         "s3_bucket_region": "us-east-1",
>         "s3_bucket_name": "my-bucket"
>         }
>     }'

3. Create a Bulk Sync

The following request creates a Bulk Sync that replicates the Salesforce Account and Contact objects into S3:

Omit schemas to sync every object on the source.

cURL

Python

Typescript

Go

$ curl --request POST \
>      --url https://app.polytomic.com/api/bulk/syncs \
>      --header "accept: application/json" \
>      --header "content-type: application/json" \
>      --header "X-Polytomic-Version: 2024-02-08" \
>      --header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
>      -d '{
>           "name": "Salesforce Bulk Sync",
>           "source_connection_id": "YOUR-SALESFORCE-CONNECTION-ID",
>           "destination_connection_id": "YOUR-S3-CONNECTION-ID",
>           "mode": "replicate",
>           "schedule": {
>               "frequency": "continuous"
>           },
>           "destination_configuration": {
>                "format": "csv"
>           },
>           "schemas": ["Account", "Contact"]
>     	}'

Bulk Sync schemas

ℹ️ Retry while schemas cache

Polytomic may take a few moments to cache source schemas after a new Connection is created. Until the cache is populated, Bulk Sync creation returns:
1 {"status":400,"message":"No schemas found. This can mean that the Polytomic has not cached any schemas yet. Please try again.","metadata":null}
This request is safe to retry until it returns a 2xx response.

$	curl --request POST \
>	--url https://app.polytomic.com/api/connections \
>	--header "accept: application/json" \
>	--header "content-type: application/json" \
>	--header "X-Polytomic-Version: 2024-02-08" \
>	--header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
>	-d '{"name": "Salesforce Connection","type": "salesforce", "configuration": {"domain": "https://example.my.salesforce.com"}}'

$	curl --request POST \
>	--url https://app.polytomic.com/api/bulk/syncs \
>	--header "accept: application/json" \
>	--header "content-type: application/json" \
>	--header "X-Polytomic-Version: 2024-02-08" \
>	--header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
>	-d '{
>	"name": "Salesforce Bulk Sync",
>	"source_connection_id": "YOUR-SALESFORCE-CONNECTION-ID",
>	"destination_connection_id": "YOUR-S3-CONNECTION-ID",
>	"mode": "replicate",
>	"schedule": {
>	"frequency": "continuous"
>	},
>	"destination_configuration": {
>	"format": "csv"
>	},
>	"schemas": ["Account", "Contact"]
>	}'