Bulk sync (ELT) from Salesforce to S3

UI demo video

You can watch a demo video of these operations in Polytomic’s admin console here.

Code

Before getting started make sure to set your Polytomic API key as an environment variable:

$export POLYTOMIC_API_KEY=YOUR-API-TOKEN

This example tutorial will cover three steps:

  1. Connecting to Salesforce.
  2. Connecting to S3.
  3. Loading data from Salesforce into S3.

1. Create a Salesforce connection

The following request will create a Salesforce connection. The configuration for each type of connection can be found at the connection configuration page.

$curl --request POST \
> --url https://app.polytomic.com/api/connections \
> --header "accept: application/json" \
> --header "content-type: application/json" \
> --header "X-Polytomic-Version: 2024-02-08" \
> --header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
> -d '{"name": "Salesforce Connection","type": "salesforce", "configuration": {"domain": "https://example.my.salesforce.com"}}'

Since Salesforce connections uses OAuth to authenticate, we’ll need to follow the link returned in the auth_url parameter in the response

Redirection


By default, the API expects the auth_url to be opened in a new browser window. If you’d like to modify the redirect behavior, there is an optional redirect_url parameter that can be added to the request body.

2. Create an S3 connection

The S3 bucket must be network-accessible from Polytomic. For more information, see our S3 Connection Guide.

$ curl --request POST \
> --url https://app.polytomic.com/api/connections \
> --header "accept: application/json" \
> --header "content-type: application/json" \
> --header "X-Polytomic-Version: 2024-02-08" \
> --header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
> -d '{
> "name": "S3 Connection",
> "type": "s3",
> "configuration": {
> "aws_access_key_id": "EXAMPLE",
> "aws_secret_access_key": "SECRET",
> "s3_bucket_region": "us-east-1",
> "s3_bucket_name": "my-bucket"
> }
> }'

3. Create a bulk sync

This will create a bulk sync of the Account and Contact Salesforce objects into S3:


Important: If the schemas parameter is omitted, all objects will be synced.


$curl --request POST \
> --url https://app.polytomic.com/api/bulk/syncs \
> --header "accept: application/json" \
> --header "content-type: application/json" \
> --header "X-Polytomic-Version: 2024-02-08" \
> --header "Authorization: Bearer ${POLYTOMIC_API_KEY}" \
> -d '{
> "name": "Salesforce Bulk Sync",
> "source_connection_id": "YOUR-SALESFORCE-CONNECTION-ID",
> "destination_connection_id": "YOUR-S3-CONNECTION-ID",
> "mode": "replicate",
> "schedule": {
> "frequency": "continuous"
> },
> "destination_configuration": {
> "format": "csv"
> },
> "schemas": ["Account", "Contact"]
> }'

Bulk sync schemas


Polytomic may take a few moments to cache source schemas after creating new connections. The message {"status":400,"message":"No schemas found. This can mean that the Polytomic has not cached any schemas yet. Please try again.","metadata":null} indicates that the schemas are not ready to be used in bulk sync configuration yet. This request is safe to retry until receiving a 2xx response.