S3

Supports:

✅ Models
✅ Model sync destination
✅ Bulk sync source
✅ Bulk sync destination

Connection

Configuration

Name	Type	Description	Required
`auth_mode`	`string`	Authentication Method How to authenticate with AWS. Defaults to Access Key and Secret. Accepted values: `access_key_and_secret` ↓, `iam_role` ↓	`true`
`csv_has_headers`	`boolean`	CSV files have headers Whether CSV files have a header row with field names.	`false`
`enable_event_notifications`	`boolean`	Enable event notifications Enable S3 event notifications for incremental sync ↓	`false`
`is_single_table`	`boolean`	Files are time-based snapshots Treat the files as a single table. ↓	`false`
`s3_bucket_name`	`string`	S3 Bucket Name Bucket name (folder optional); ex: s3://polytomic/dataset	`true`
`s3_bucket_region`	`string`	S3 Bucket Region	`true`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

Name	Type	Description	Required
`aws_access_key_id`	`string`	AWS Access Key ID Access Key ID with read/write access to a bucket.	`true`
`aws_secret_access_key`	`string`	AWS Secret Access Key	`true`

When `auth_mode` is `iam_role`

Name	Type	Description	Required
`iam_role_arn`	`string`	IAM Role ARN	`true`

`enable_event_notifications`

When `enable_event_notifications` is `true`

Name	Type	Description	Required
`event_queue_arn`	`string`	Event Queue ARN ARN of the SQS queue receiving S3 event notifications	`true`

`is_single_table`

When `is_single_table` is `true`

Name	Type	Description	Required
`is_directory_snapshot`	`boolean`	Multi-directory multi-table	`false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `false`

Name	Type	Description	Required
`single_table_file_format`	`string`	File format Accepted values: `csv`, `json`, `parquet`	`false`
`single_table_name`	`string`	Collection name	`false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `true`

Name	Type	Description	Required
`single_table_file_formats`	`array`	File formats that may be present across different tables	`false`

`is_directory_snapshot`

When `is_directory_snapshot` is `true`

Name	Type	Description	Required
`directory_glob_pattern`	`string`	Tables glob path	`true`

`single_table_file_format`

When `single_table_file_format` is `csv` OR `single_table_file_formats` contains `csv`

Name	Type	Description	Required
`skip_lines`	`integer`	Skip first lines Skip first N lines of each CSV file.	`false`

Example

1 {
2   "name": "S3 connection",
3   "type": "s3",
4   "configuration": {
5     "auth_mode": "access_key_and_secret",
6     "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7     "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8     "csv_has_headers": false,
9     "enable_event_notifications": false,
10     "is_single_table": false,
11     "s3_bucket_name": "s3://polytomic/dataset",
12     "s3_bucket_region": "us-east-1"
13   }
14 }

Read-only properties

Name	Type	Description	Required

`auth_mode`

When `auth_mode` is `access_key_and_secret`

Name	Type	Description	Required
`aws_user`	`string`	User ARN	`false`

When `auth_mode` is `iam_role`

Name	Type	Description	Required
`external_id`	`string`	External ID for the IAM role	`false`

Model Sync

Source

Configuration

Name	Type	Description	Required
`file_format`	`string`	File format Accepted values: `csv`, `json`, `parquet`	`false`
`key`	`string`	Object key The key of the object in the bucket to read from.	`false`
`model_from`	`string`	Files The model is generated from a single file or a multi-file archive. Accepted values: `single_file`, `multi_file_archive`	`true`
`skip_lines`	`integer`	Skip first lines Skip first N lines of each CSV file.	`false`
`subfolder`	`string`	Subfolder to read files from from (optional)	`false`

Example

1 {
2   ...
3   "configuration": {
4     "file_format": "",
5     "key": "",
6     "model_from": "",
7     "skip_lines": 0,
8     "subfolder": ""
9   }
10 }

Target

S3 connections may be used as the destination in a model sync.

All targets

Configuration

Name	Type	Description	Required
`format`	`string`	Output format Output file encoding. Accepted values: `csv`, `json-doc`, `json`, `parquet`	`false`

Example

1 {
2   ...
3   "target": {
4       "configuration": {
5         "format": "csv"
6     }
7   }
8 }

Bulk Sync

Source

S3 connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

Name	Type	Description	Required
`format`	`string`	Output file encoding	`false`
`subfolder`	`string`	Subfolder to write to (optional)	`false`

Example

1 {
2   ...
3   "destination_configuration": {
4     "format": "csv",
5     "subfolder": "reports"
6   }
7 }

1	{
2	"name": "S3 connection",
3	"type": "s3",
4	"configuration": {
5	"auth_mode": "access_key_and_secret",
6	"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7	"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8	"csv_has_headers": false,
9	"enable_event_notifications": false,
10	"is_single_table": false,
11	"s3_bucket_name": "s3://polytomic/dataset",
12	"s3_bucket_region": "us-east-1"
13	}
14	}

1	{
2	...
3	"configuration": {
4	"file_format": "",
5	"key": "",
6	"model_from": "",
7	"skip_lines": 0,
8	"subfolder": ""
9	}
10	}

1	{
2	...
3	"destination_configuration": {
4	"format": "csv",
5	"subfolder": "reports"
6	}
7	}

Connection

Configuration

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

enable_event_notifications

When enable_event_notifications is true

is_single_table

When is_single_table is true

When is_single_table is true AND is_directory_snapshot is false

When is_single_table is true AND is_directory_snapshot is true

is_directory_snapshot

When is_directory_snapshot is true

single_table_file_format

When single_table_file_format is csv OR single_table_file_formats contains csv

Example

Read-only properties

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

Model Sync

Source

Configuration

Example

Target

All targets

Configuration

Example

Bulk Sync

Source

Destination

Configuration

Example

Connection

Configuration

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

enable_event_notifications

When enable_event_notifications is true

is_single_table

When is_single_table is true

When is_single_table is true AND is_directory_snapshot is false

When is_single_table is true AND is_directory_snapshot is true

is_directory_snapshot

When is_directory_snapshot is true

single_table_file_format

When single_table_file_format is csv OR single_table_file_formats contains csv

Example

Read-only properties

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

Model Sync

Source

Configuration

Example

Target

All targets

Configuration

Example

Bulk Sync

Source

Destination

Configuration

Example

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`

`enable_event_notifications`

When `enable_event_notifications` is `true`

`is_single_table`

When `is_single_table` is `true`

When `is_single_table` is `true` AND `is_directory_snapshot` is `false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `true`

`is_directory_snapshot`

When `is_directory_snapshot` is `true`

`single_table_file_format`

When `single_table_file_format` is `csv` OR `single_table_file_formats` contains `csv`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`

`enable_event_notifications`

When `enable_event_notifications` is `true`

`is_single_table`

When `is_single_table` is `true`

When `is_single_table` is `true` AND `is_directory_snapshot` is `false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `true`

`is_directory_snapshot`

When `is_directory_snapshot` is `true`

`single_table_file_format`

When `single_table_file_format` is `csv` OR `single_table_file_formats` contains `csv`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`