Connection

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
access_idstringtruefalse
secret_access_keystringtruefalse
aws_userstringfalsetrue
regionstringtruefalse
outputbucketstringA pre-existing bucket (folder optional) that AWS can use to store query results. ex: s3://polytomic-athena-results/customer-datasettruefalse

Example

1{
2 "name": "awsathena connection",
3 "type": "awsathena",
4 "configuration": {
5 "access_id": "AKIAIOSFODNN7EXAMPLE",
6 "outputbucket": "s3://polytomic-athena-results/customer-dataset",
7 "region": "us-east-1",
8 "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
9 }
10}

Model Sync

Source

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
querystringfalsefalse
catalogstringCatalogfalsefalse
databasestringDatabasefalsefalse
tablestringTablefalsefalse

Example

1{
2 ...
3 "configuration": {
4 "catalog": "awsdatacatalog",
5 "database": "default",
6 "query": "SELECT * FROM \"webdata\".\"impressions\"",
7 "table": "users"
8 }
9}

Target

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
preserve_table_on_resyncbooleanPreserve destination table when resyncingfalsefalse

Example

1{
2 ...
3 "target": {
4 "configuration": {
5 "preserve_table_on_resync": false
6 }
7 }
8}

Bulk Sync

Destination

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
advancedobjectfalsefalse
schemastringOutput schema to write tofalsefalse
mirror_schemasbooleanIndicates whether schemas should be mirroredfalsefalse

Example

1{
2 ...
3 "destination_configuration": {
4 "advanced": {
5 "table_prefix": "",
6 "truncate_existing": false
7 },
8 "mirror_schemas": false,
9 "schema": "public"
10 }
11}

Type handling

Destination types

POLYTOMIC TYPEAWS ATHENA TYPE
array<>string
bigintstring
booleanboolean
datetimestamp
datetimetimestamp
decimal(precision, scale)string
doublestring
intstring
jsonstring
jsonarraystring
numberstring
object{}string
singlestring
smallintstring
stringstring
timetimestamp

Source types

AWS ATHENA TYPEPOLYTOMIC TYPE
ARRAYjsonarray
BIGINTbigint
BINARYstring
CHARstring
DATEdate
DECIMALnumber
DECIMAL(precision, scale)decimal(precision, scale)
DOUBLEnumber
FLOATnumber
INTint
INTEGERint
INTERVAL DAY TO SECONDstring
INTERVAL YEAR TO MONTHstring
IPADDRESSstring
JSONjson
MAPjson
REALnumber
SMALLINTsmallint
STRINGstring
TIMEtime
TIME WITH TIME ZONEtime
TIMESTAMPdatetime
TIMESTAMP WITH TIME ZONEdatetime
TINYINTsmallint
VARBINARYstring
VARCHARstring