Skip to navigation
Configuring your connectionsConnection configurations

S3 Compatible

Supports:

  • ✅ Models
  • ✅ Model sync destination
  • ✅ Bulk sync source
  • ✅ Bulk sync destination

Connection

Configuration

NameTypeDescriptionRequired
aws_access_key_idstringAccess key ID with read/write access to the bucket.required
aws_secret_access_keystringSecret access keyrequired
csv_has_headersbooleanCSV files have headers

Whether CSV files have a header row with field names.		optional
endpointstringEndpoint

URL of the S3-compatible API; ex: https://s3.example.com		required
is_single_tablebooleanFiles are time-based snapshots

Treat the files as a single table. ↓		optional
s3_bucket_namestringS3 bucket name

Bucket name (folder optional); ex: s3://polytomic/dataset		required
s3_bucket_regionstringS3 bucket region

Region to sign requests for. Most S3-compatible stores ignore this value.		optional

is_single_table = true

NameTypeDescriptionRequired
is_directory_snapshotbooleanMulti-directory multi-table ↓optional
{
  "name": "S3 Compatible connection",
  "type": "s3_compatible",
  "configuration": {
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "endpoint": "https://s3.example.com",
    "is_directory_snapshot": false,
    "is_single_table": true,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1",
    "single_table_file_format": "csv",
    "single_table_name": "collection",
    "skip_lines": 0
  }
}
is_directory_snapshot

When is_directory_snapshot is true:

NameTypeDescriptionRequired
directory_glob_patternstringTables glob pathrequired
{
  "name": "S3 Compatible connection",
  "type": "s3_compatible",
  "configuration": {
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "directory_glob_pattern": "",
    "endpoint": "https://s3.example.com",
    "is_directory_snapshot": true,
    "is_single_table": true,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1",
    "single_table_file_formats": []
  }
}

is_single_table = true and is_directory_snapshot = false

NameTypeDescriptionRequired
single_table_file_formatstringFile format

Accepted values: csv ↓, json, parquet		optional
single_table_namestringCollection nameoptional
single_table_file_format

When single_table_file_format is csv or single_table_file_formats contains csv:

NameTypeDescriptionRequired
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.		optional

is_single_table = true and is_directory_snapshot = true

NameTypeDescriptionRequired
single_table_file_formatsarrayFile formats that may be present across different tablesoptional

Example

{
  "name": "S3 Compatible connection",
  "type": "s3_compatible",
  "configuration": {
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "csv_has_headers": true,
    "endpoint": "https://s3.example.com",
    "is_single_table": false,
    "s3_bucket_name": "s3://polytomic/dataset",
    "s3_bucket_region": "us-east-1"
  }
}

Model Sync

Source

Configuration

NameTypeDescriptionRequired
file_formatstringFile format

Accepted values: csv, json, parquet		optional
keystringObject key

The key of the object in the bucket to read from.		optional
model_fromstringFiles

The model is generated from a single file or a multi-file archive. Accepted values: single_file, multi_file_archive		required
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.		optional
subfolderstringSubfolder to read files from from (optional)optional

Example

{
  ...
  "configuration": {
    "file_format": "csv",
    "key": "",
    "model_from": "single_file",
    "skip_lines": 0,
    "subfolder": ""
  }
}

Target

S3 Compatible connections may be used as the destination in a model sync.

All targets

Configuration
NameTypeDescriptionRequired
formatstringOutput format

Output file encoding. Accepted values: csv, json-doc, json, parquet		optional
Example
{
  ...
  "target": {
      "configuration": {
        "format": "csv"
    }
  }
}

Bulk Sync

Source

S3 Compatible connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

NameTypeDescriptionRequired
advancedobjectoptional
formatstringOutput file encodingoptional
subfolderstringSubfolder to write to (optional)optional

Advanced configuration (advanced)

NameTypeDescriptionRequired
write_incrementalbooleanWrite incremental files when possibleoptional

Example

{
  ...
  "destination_configuration": {
    "advanced": {
        "write_incremental": false
    },
    "format": "csv",
    "subfolder": "reports"
  }
}