Cloudflare R2 | Polytomic Docs

Supports:

✅ Models
✅ Model sync destination
✅ Bulk sync source
✅ Bulk sync destination

Connection

Configuration

Name	Type	Description	Required
`account_id`	`string`	Account ID	`true`
`aws_access_key_id`	`string`	Access Key ID with read/write access to a bucket.	`true`
`aws_secret_access_key`	`string`	Secret Access Key	`true`
`bucket_name`	`string`	Bucket name (folder optional); ex: polytomic/dataset	`true`
`csv_has_headers`	`boolean`	CSV files have headers Whether CSV files have a header row with field names.	`false`
`is_single_table`	`boolean`	Files are time-based snapshots Treat the files as a single table. ↓	`false`

`is_single_table`

When `is_single_table` is `true`

Name	Type	Description	Required
`is_directory_snapshot`	`boolean`	Multi-directory multi-table	`false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `false`

Name	Type	Description	Required
`single_table_file_format`	`string`	File format Accepted values: `csv`, `json`, `parquet`	`false`
`single_table_name`	`string`	Collection name	`false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `true`

Name	Type	Description	Required
`single_table_file_formats`	`array`	File formats that may be present across different tables	`false`

`is_directory_snapshot`

When `is_directory_snapshot` is `true`

Name	Type	Description	Required
`directory_glob_pattern`	`string`	Tables glob path	`true`

`single_table_file_format`

When `single_table_file_format` is `csv` OR `single_table_file_formats` contains `csv`

Name	Type	Description	Required
`skip_lines`	`integer`	Skip first lines Skip first N lines of each CSV file.	`false`

Example

1 {
2   "name": "Cloudflare R2 connection",
3   "type": "cloudflare_r2",
4   "configuration": {
5     "account_id": "",
6     "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
7     "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
8     "bucket_name": "polytomic/dataset",
9     "csv_has_headers": false,
10     "is_single_table": false
11   }
12 }

Model Sync

Source

Configuration

Name	Type	Description	Required
`file_format`	`string`	File format Accepted values: `csv`, `json`, `parquet`	`false`
`key`	`string`	Object key The key of the object in the bucket to read from.	`false`
`model_from`	`string`	Files The model is generated from a single file or a multi-file archive. Accepted values: `single_file`, `multi_file_archive`	`true`
`skip_lines`	`integer`	Skip first lines Skip first N lines of each CSV file.	`false`
`subfolder`	`string`	Subfolder to read files from from (optional)	`false`

Example

1 {
2   ...
3   "configuration": {
4     "file_format": "",
5     "key": "",
6     "model_from": "",
7     "skip_lines": 0,
8     "subfolder": ""
9   }
10 }

Target

Cloudflare R2 connections may be used as the destination in a model sync.

All targets

Configuration

Name	Type	Description	Required
`format`	`string`	Output format Output file encoding. Accepted values: `csv`, `json-doc`, `json`, `parquet`	`false`

Example

1 {
2   ...
3   "target": {
4       "configuration": {
5         "format": "csv"
6     }
7   }
8 }

Bulk Sync

Destination

Configuration

Name	Type	Description	Required
`format`	`string`	Output file encoding	`false`
`subfolder`	`string`	Subfolder to write to (optional)	`false`

Example

1 {
2   ...
3   "destination_configuration": {
4     "format": "csv",
5     "subfolder": "reports"
6   }
7 }

Connection

Configuration

is_single_table

When is_single_table is true

When is_single_table is true AND is_directory_snapshot is false

When is_single_table is true AND is_directory_snapshot is true

is_directory_snapshot

When is_directory_snapshot is true

single_table_file_format

When single_table_file_format is csv OR single_table_file_formats contains csv

Example

Model Sync

Source

Configuration

Example

Target

All targets

Configuration

Example

Bulk Sync

Destination

Configuration

Example

`is_single_table`

When `is_single_table` is `true`

When `is_single_table` is `true` AND `is_directory_snapshot` is `false`

When `is_single_table` is `true` AND `is_directory_snapshot` is `true`

`is_directory_snapshot`

When `is_directory_snapshot` is `true`

`single_table_file_format`

When `single_table_file_format` is `csv` OR `single_table_file_formats` contains `csv`