S3 | Polytomic Docs

Supports:

✅ Models
✅ Model sync destination
✅ Bulk sync source
✅ Bulk sync destination

Connection

Configuration

Name	Type	Description	Required
`auth_mode`	`string`	Authentication method How to authenticate with AWS. Defaults to Access Key and Secret. Accepted values: `access_key_and_secret` ↓, `iam_role` ↓	`required`
`csv_has_headers`	`boolean`	CSV files have headers Whether CSV files have a header row with field names.	`optional`
`enable_event_notifications`	`boolean`	Enable event notifications Enable S3 event notifications for incremental sync ↓	`optional`
`is_single_table`	`boolean`	Files are time-based snapshots Treat the files as a single table. ↓	`optional`
`s3_bucket_name`	`string`	S3 bucket name Bucket name (folder optional); ex: s3://polytomic/dataset	`required`
`s3_bucket_region`	`string`	S3 bucket region	`required`

`auth_mode`

When auth_mode is access_key_and_secret:

Name	Type	Description	Required
`aws_access_key_id`	`string`	AWS access key ID Access Key ID with read/write access to a bucket.	`required`
`aws_secret_access_key`	`string`	AWS secret access key	`required`

Read-only property	Type	Description
`aws_user`	`string`	User ARN

{
  "name": "S3 connection",
  "type": "s3",
  "configuration": {
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "enable_event_notifications": false,
    "is_single_table": false,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1"
  }
}

When auth_mode is iam_role:

Name	Type	Description	Required
`iam_role_arn`	`string`	IAM role ARN	`required`

Read-only property	Type	Description
`external_id`	`string`	External ID for the IAM role

{
  "name": "S3 connection",
  "type": "s3",
  "configuration": {
    "auth_mode": "iam_role",
    "csv_has_headers": true,
    "enable_event_notifications": false,
    "iam_role_arn": "",
    "is_single_table": false,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1"
  }
}

`enable_event_notifications`

When enable_event_notifications is true:

Name	Type	Description	Required
`event_queue_arn`	`string`	Event queue ARN ARN of the SQS queue receiving S3 event notifications	`required`
`propagate_file_deletions`	`boolean`	Propagate file deletion events When a file is deleted from the bucket, soft-delete the corresponding records in supported destinations on the next sync. Requires the record key to be captured from the file path.	`optional`

{
  "name": "S3 connection",
  "type": "s3",
  "configuration": {
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "enable_event_notifications": true,
    "event_queue_arn": "arn:aws:sqs:us-east-1:123456789012:my-queue",
    "is_single_table": false,
    "propagate_file_deletions": false,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1"
  }
}

`is_single_table` = `true`

Name	Type	Description	Required
`is_directory_snapshot`	`boolean`	Multi-directory multi-table ↓	`optional`

{
  "name": "S3 connection",
  "type": "s3",
  "configuration": {
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "enable_event_notifications": false,
    "is_directory_snapshot": false,
    "is_single_table": true,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1",
    "single_table_file_format": "csv",
    "single_table_name": "collection",
    "skip_lines": 0
  }
}

`is_directory_snapshot`

When is_directory_snapshot is true:

Name	Type	Description	Required
`directory_glob_pattern`	`string`	Tables glob path	`required`

{
  "name": "S3 connection",
  "type": "s3",
  "configuration": {
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "directory_glob_pattern": "",
    "enable_event_notifications": false,
    "is_directory_snapshot": true,
    "is_single_table": true,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1",
    "single_table_file_formats": []
  }
}

`is_single_table` = `true` and `is_directory_snapshot` = `false`

Name	Type	Description	Required
`single_table_file_format`	`string`	File format Accepted values: `csv` ↓, `json`, `parquet`	`optional`
`single_table_name`	`string`	Collection name	`optional`

`single_table_file_format`

When single_table_file_format is csv or single_table_file_formats contains csv:

Name	Type	Description	Required
`skip_lines`	`integer`	Skip first lines Skip first N lines of each CSV file.	`optional`

`is_single_table` = `true` and `is_directory_snapshot` = `true`

Name	Type	Description	Required
`single_table_file_formats`	`array`	File formats that may be present across different tables	`optional`

Model Sync

Source

Configuration

Name	Type	Description	Required
`file_format`	`string`	File format Accepted values: `csv`, `json`, `parquet`	`optional`
`key`	`string`	Object key The key of the object in the bucket to read from.	`optional`
`model_from`	`string`	Files The model is generated from a single file or a multi-file archive. Accepted values: `single_file`, `multi_file_archive`	`required`
`skip_lines`	`integer`	Skip first lines Skip first N lines of each CSV file.	`optional`
`subfolder`	`string`	Subfolder to read files from from (optional)	`optional`

Example

{
  ...
  "configuration": {
    "file_format": "csv",
    "key": "",
    "model_from": "single_file",
    "skip_lines": 0,
    "subfolder": ""
  }
}

Target

S3 connections may be used as the destination in a model sync.

All targets

Configuration

Name	Type	Description	Required
`format`	`string`	Output format Output file encoding. Accepted values: `csv`, `json-doc`, `json`, `parquet`	`optional`

Example

{
  ...
  "target": {
      "configuration": {
        "format": "csv"
    }
  }
}

Bulk Sync

Source

S3 connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

Name	Type	Description	Required
`advanced`	`object`		`optional`
`format`	`string`	Output file encoding	`optional`
`subfolder`	`string`	Subfolder to write to (optional)	`optional`

Example

{
  ...
  "destination_configuration": {
    "advanced": {
        "write_incremental": false
    },
    "format": "csv",
    "subfolder": "reports"
  }
}

Connection

Configuration

auth_mode

enable_event_notifications

is_single_table = true

is_directory_snapshot

is_single_table = true and is_directory_snapshot = false

single_table_file_format

is_single_table = true and is_directory_snapshot = true

Model Sync

Source

Configuration

Example

Target

All targets

Configuration

Example

Bulk Sync

Source

Destination

Configuration

Example

`auth_mode`

`enable_event_notifications`

`is_single_table` = `true`

`is_directory_snapshot`

`is_single_table` = `true` and `is_directory_snapshot` = `false`

`single_table_file_format`

`is_single_table` = `true` and `is_directory_snapshot` = `true`