AWS Athena — Polytomic Docs

Connection

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
access_id	string		true	false
secret_access_key	string		true	false
aws_user	string		false	true
region	string		true	false
outputbucket	string	A pre-existing bucket (folder optional) that AWS can use to store query results. ex: s3://polytomic-athena-results/customer-dataset	true	false

Example

1 {
2   "name": "awsathena connection",
3   "type": "awsathena",
4   "configuration": {
5     "access_id": "AKIAIOSFODNN7EXAMPLE",
6     "outputbucket": "s3://polytomic-athena-results/customer-dataset",
7     "region": "us-east-1",
8     "secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
9   }
10 }

Model Sync

Source

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
query	string		false	false
catalog	string	Catalog	false	false
database	string	Database	false	false
table	string	Table	false	false

Example

1 {
2   ...
3   "configuration": {
4     "catalog": "awsdatacatalog",
5     "database": "default",
6     "query": "SELECT * FROM \"webdata\".\"impressions\"",
7     "table": "users"
8   }
9 }

Target

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
preserve_table_on_resync	boolean	Preserve destination table when resyncing	false	false
write_record_timestamps	boolean	Write row timestamp metadata	false	false
created_column	string	’Created at’ timestamp column	false	false
updated_column	string	’Updated at’ timestamp column	false	false

Example

1 {
2   ...
3   "target": {
4       "configuration": {
5         "created_column": "",
6         "preserve_table_on_resync": false,
7         "updated_column": "",
8         "write_record_timestamps": false
9     }
10   }
11 }

Bulk Sync

Destination

Configuration

NAME	TYPE	DESCRIPTION	REQUIRED	READONLY
advanced	object		false	false
mirror_schemas	boolean	Indicates whether schemas should be mirrored	false	false
schema	string	Output schema to write to	false	false

Example

1 {
2   ...
3   "destination_configuration": {
4     "advanced": {
5         "table_prefix": "",
6         "truncate_existing": false
7     },
8     "mirror_schemas": false,
9     "schema": "public"
10   }
11 }

Type handling

Destination types

POLYTOMIC TYPE	AWS ATHENA TYPE
`array<>`	`string`
`bigint`	`string`
`boolean`	`boolean`
`date`	`timestamp`
`datetime`	`timestamp`
`decimal(precision, scale)`	`string`
`double`	`string`
`int`	`string`
`json`	`string`
`jsonarray`	`string`
`number`	`string`
`object{}`	`string`
`single`	`string`
`smallint`	`string`
`string`	`string`
`time`	`timestamp`

Source types

AWS ATHENA TYPE	POLYTOMIC TYPE
`ARRAY`	`jsonarray`
`BIGINT`	`bigint`
`BINARY`	`string`
`CHAR`	`string`
`DATE`	`date`
`DECIMAL`	`number`
`DECIMAL(precision, scale)`	`decimal(precision, scale)`
`DOUBLE`	`number`
`FLOAT`	`number`
`INT`	`int`
`INTEGER`	`int`
`INTERVAL DAY TO SECOND`	`string`
`INTERVAL YEAR TO MONTH`	`string`
`IPADDRESS`	`string`
`JSON`	`json`
`MAP`	`json`
`REAL`	`number`
`SMALLINT`	`smallint`
`STRING`	`string`
`TIME`	`time`
`TIME WITH TIME ZONE`	`time`
`TIMESTAMP`	`datetime`
`TIMESTAMP WITH TIME ZONE`	`datetime`
`TINYINT`	`smallint`
`VARBINARY`	`string`
`VARCHAR`	`string`

1	{
2	"name": "awsathena connection",
3	"type": "awsathena",
4	"configuration": {
5	"access_id": "AKIAIOSFODNN7EXAMPLE",
6	"outputbucket": "s3://polytomic-athena-results/customer-dataset",
7	"region": "us-east-1",
8	"secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
9	}
10	}

1	{
2	...
3	"configuration": {
4	"catalog": "awsdatacatalog",
5	"database": "default",
6	"query": "SELECT * FROM \"webdata\".\"impressions\"",
7	"table": "users"
8	}
9	}

1	{
2	...
3	"target": {
4	"configuration": {
5	"created_column": "",
6	"preserve_table_on_resync": false,
7	"updated_column": "",
8	"write_record_timestamps": false
9	}
10	}
11	}

1	{
2	...
3	"destination_configuration": {
4	"advanced": {
5	"table_prefix": "",
6	"truncate_existing": false
7	},
8	"mirror_schemas": false,
9	"schema": "public"
10	}
11	}