Connection

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
auth_modestringHow to authenticate with AWS. Defaults to Access Key and Secret. Accepted Values: access_key_and_secret, iam_roletruefalse
aws_access_key_idstringAccess Key ID with read/write access to a bucket. (required if auth_mode is “aws_access_key_and_secret”)falsefalse
aws_secret_access_keystring(required if auth_mode is “aws_access_key_and_secret”)falsefalse
aws_userstringfalsefalse
iam_role_arnstring(required if auth_mode is “iam_role”)falsefalse
external_idstringExternal ID for the IAM rolefalsefalse
s3_bucket_regionstringtruefalse
s3_bucket_namestringBucket name (folder optional); ex: s3://polytomic/datasettruefalse
is_single_tablebooleanTreat the files as a single table. (required if “ is “true”)falsefalse
single_table_namestring(required if is_single_table is “true”)falsefalse
single_table_file_formatstring(required if is_single_table is “true”). Accepted Values: csv, jsonfalsefalse
skip_linesintegerSkip first N lines of each CSV file.falsefalse

Example

1{
2 "name": "s3 connection",
3 "type": "s3",
4 "configuration": {
5 "auth_mode": "access_key_and_secret",
6 "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7 "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8 "aws_user": "",
9 "external_id": "",
10 "iam_role_arn": "",
11 "is_single_table": false,
12 "s3_bucket_name": "s3://polytomic/dataset",
13 "s3_bucket_region": "us-east-1",
14 "single_table_file_format": "",
15 "single_table_name": "collection",
16 "skip_lines": 0
17 }
18}

Model Sync

Source

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
model_fromstringThe model is generated from a single file or a multi-file archive. Accepted Values: single_file, multi_file_archivetruefalse
file_formatstringFile format Accepted Values: csv, jsonfalsefalse
skip_linesintegerSkip first N lines of each CSV file.falsefalse
keystringThe key of the object in the bucket to read from.falsefalse
subfolderstringSubfolder to read files from from (optional)falsefalse

Example

1{
2 ...
3 "configuration": {
4 "file_format": "",
5 "key": "",
6 "model_from": "",
7 "skip_lines": 0,
8 "subfolder": ""
9 }
10}

Target

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
formatstringOutput file encodingfalsefalse

Example

1{
2 ...
3 "target": {
4 "configuration": {
5 "format": "csv"
6 }
7 }
8}

Bulk Sync

Destination

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
subfolderstringSubfolder to write to (optional)falsefalse
formatstringOutput file encodingfalsefalse

Example

1{
2 ...
3 "destination_configuration": {
4 "format": "csv",
5 "subfolder": "reports"
6 }
7}