S3 (2021-05-23) — Polytomic Docs

Connection

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
auth_mode	string	How to authenticate with AWS. Defaults to Access Key and Secret. Accepted Values: access_key_and_secret, iam_role	true	false
aws_access_key_id	string	Access Key ID with read/write access to a bucket. (required if `auth_mode` is “aws_access_key_and_secret”)	false	false
aws_secret_access_key	string	(required if `auth_mode` is “aws_access_key_and_secret”)	false	false
aws_user	string		false	false
iam_role_arn	string	(required if `auth_mode` is “iam_role”)	false	false
external_id	string	External ID for the IAM role	false	false
s3_bucket_region	string		true	false
s3_bucket_name	string	Bucket name (folder optional); ex: s3://polytomic/dataset	true	false
is_single_table	boolean	Treat the files as a single table. (required if “ is “true”)	false	false
single_table_name	string	(required if `is_single_table` is “true”)	false	false
single_table_file_format	string	(required if `is_single_table` is “true”). Accepted Values: csv, json	false	false
skip_lines	integer	Skip first N lines of each CSV file.	false	false

Example

1 {
2   "name": "s3 connection",
3   "type": "s3",
4   "configuration": {
5     "auth_mode": "access_key_and_secret",
6     "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7     "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8     "aws_user": "",
9     "external_id": "",
10     "iam_role_arn": "",
11     "is_single_table": false,
12     "s3_bucket_name": "s3://polytomic/dataset",
13     "s3_bucket_region": "us-east-1",
14     "single_table_file_format": "",
15     "single_table_name": "collection",
16     "skip_lines": 0
17   }
18 }

Model Sync

Source

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
model_from	string	The model is generated from a single file or a multi-file archive. Accepted Values: single_file, multi_file_archive	true	false
file_format	string	File format Accepted Values: csv, json	false	false
skip_lines	integer	Skip first N lines of each CSV file.	false	false
key	string	The key of the object in the bucket to read from.	false	false
subfolder	string	Subfolder to read files from from (optional)	false	false

Example

1 {
2   ...
3   "configuration": {
4     "file_format": "",
5     "key": "",
6     "model_from": "",
7     "skip_lines": 0,
8     "subfolder": ""
9   }
10 }

Target

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
format	string	Output file encoding	false	false

Example

1 {
2   ...
3   "target": {
4       "configuration": {
5         "format": "csv"
6     }
7   }
8 }

Bulk Sync

Destination

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
subfolder	string	Subfolder to write to (optional)	false	false
format	string	Output file encoding	false	false

Example

1 {
2   ...
3   "destination_configuration": {
4     "format": "csv",
5     "subfolder": "reports"
6   }
7 }

1	{
2	"name": "s3 connection",
3	"type": "s3",
4	"configuration": {
5	"auth_mode": "access_key_and_secret",
6	"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7	"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8	"aws_user": "",
9	"external_id": "",
10	"iam_role_arn": "",
11	"is_single_table": false,
12	"s3_bucket_name": "s3://polytomic/dataset",
13	"s3_bucket_region": "us-east-1",
14	"single_table_file_format": "",
15	"single_table_name": "collection",
16	"skip_lines": 0
17	}
18	}

1	{
2	...
3	"configuration": {
4	"file_format": "",
5	"key": "",
6	"model_from": "",
7	"skip_lines": 0,
8	"subfolder": ""
9	}
10	}

1	{
2	...
3	"destination_configuration": {
4	"format": "csv",
5	"subfolder": "reports"
6	}
7	}