Configuring your connectionsConnection configurations

S3

Connection

Description

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
auth_modestringHow to authenticate with AWS. Defaults to Access Key and Secrettruefalse
aws_access_key_idstringAccess Key ID with read/write access to a bucket. (required if AuthMode is “aws_access_key_and_secret”)falsefalse
aws_secret_access_keystring(required if AuthMode is “aws_access_key_and_secret”)falsefalse
aws_userstringfalsefalse
iam_role_arnstring(required if AuthMode is “iam_role”)falsefalse
external_idstringExternal ID for the IAM rolefalsefalse
s3_bucket_regionstringtruefalse
s3_bucket_namestringBucket name (folder optional); ex: s3://polytomic/datasettruefalse
is_single_tablebooleanTreat the files as a single table. (required if is_single_table is “true”)falsefalse
single_table_namestring(required if IsSingleTable is “true”)falsefalse
single_table_file_formatstring(required if IsSingleTable is “true”)falsefalse
skip_linesintegerSkip first N lines of each CSV file.falsefalse

Example

1{
2 "name": "s3 connection",
3 "type": "s3",
4 "configuration": {
5 "auth_mode": "access_key_and_secret",
6 "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7 "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8 "aws_user": "",
9 "external_id": "",
10 "iam_role_arn": "",
11 "is_single_table": false,
12 "s3_bucket_name": "s3://polytomic/dataset",
13 "s3_bucket_region": "us-east-1",
14 "single_table_file_format": "",
15 "single_table_name": "collection",
16 "skip_lines": 0
17 }
18}

Model Sync

Source

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
keystringThe key of the object in the bucket to read from.falsefalse

Example

1{
2 ...
3 "configuration": {
4 "key": ""
5 }
6}

Target

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
formatstringOutput file encodingfalsefalse

Example

1{
2 ...
3 "target": {
4 "configuration": {
5 "format": "csv"
6 }
7 }
8}

Bulk Sync

Source

N/A

Destination

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
subfolderstringSubfolder to write to (optional)falsefalse
formatstringOutput file encodingfalsefalse

Example

1{
2 ...
3 "destination_configuration": {
4 "format": "csv",
5 "subfolder": "reports"
6 }
7}