Connection
Configuration
NAME | TYPE | DESCRIPTION | REQUIRED | READONLY |
---|---|---|---|---|
server_hostname | string | true | false | |
port | integer | true | false | |
access_token | string | true | false | |
http_path | string | true | false | |
cloud_provider | string | Accepted Values: aws, azure | false | false |
auth_mode | string | How to authenticate with AWS. Defaults to Access Key and Secret. Accepted Values: access_key_and_secret, iam_role | true | false |
iam_role_arn | string | (required if auth_mode is “iam_role”) | false | false |
storage_credential_name | string | false | false | |
external_id | string | External ID for the IAM role | false | false |
aws_access_key_id | string | See https://docs.polytomic.com/docs/databricks-connections#writing-to-databricks (required if auth_mode is “aws_access_key_id”) | false | false |
aws_secret_access_key | string | (required if auth_mode is “aws_access_key_id”) | false | false |
aws_user | string | false | false | |
s3_bucket_name | string | Name of bucket used for staging data load files (required if cloud_provider is “aws”) | false | false |
s3_bucket_region | string | (required if cloud_provider is “aws”) | false | false |
azure_account_name | string | The account name of the storage account (required if cloud_provider is “azure”) | false | false |
azure_access_key | string | The access key associated with this storage account (required if cloud_provider is “azure”) | false | false |
container_name | string | The container which we will stage files in (required if cloud_provider is “azure”) | false | false |
unity_catalog_enabled | boolean | false | false | |
enable_delta_uniform | boolean | false | false | |
enforce_query_limit | boolean | false | false | |
concurrent_queries | integer | false | false |
Example
1 { 2 "name": "databricks connection", 3 "type": "databricks", 4 "configuration": { 5 "access_token": "isoz8af6zvp8067gu68gvrp0oftevn", 6 "auth_mode": "access_key_and_secret", 7 "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE", 8 "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY", 9 "aws_user": "", 10 "azure_access_key": "abcdefghijklmnopqrstuvwxyz0123456789/+ABCDEabcdefghijklmnopqrstuvwxyz0123456789/+ABCDE==", 11 "azure_account_name": "account", 12 "cloud_provider": "aws", 13 "concurrent_queries": 0, 14 "container_name": "container", 15 "enable_delta_uniform": false, 16 "enforce_query_limit": false, 17 "external_id": "", 18 "http_path": "/sql", 19 "iam_role_arn": "", 20 "port": 443, 21 "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset", 22 "s3_bucket_region": "", 23 "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com", 24 "storage_credential_name": "", 25 "unity_catalog_enabled": false 26 } 27 }
Model Sync
Source
Configuration
NAME | TYPE | DESCRIPTION | REQUIRED | READONLY |
---|---|---|---|---|
catalog | string | Catalog | false | false |
schema | string | Schema | false | false |
table | string | Table | false | false |
query | string | Query | false | false |
Example
1 { 2 ... 3 "configuration": { 4 "catalog": "samples", 5 "query": "SELECT * FROM samples.nyctaxi.trips", 6 "schema": "nyctaxi", 7 "table": "trips" 8 } 9 }
Target
Configuration
NAME | TYPE | DESCRIPTION | REQUIRED | READONLY |
---|---|---|---|---|
preserve_table_on_resync | boolean | Preserve destination table when resyncing | false | false |
Example
1 { 2 ... 3 "target": { 4 "configuration": { 5 "preserve_table_on_resync": false 6 } 7 } 8 }
Bulk Sync
Destination
Configuration
NAME | TYPE | DESCRIPTION | REQUIRED | READONLY |
---|---|---|---|---|
advanced | object | false | false | |
catalog | string | Catalog | false | false |
schema | string | Output schema | false | false |
mirror_schemas | boolean | Mirror schemas | false | false |
external_location_name | string | External location | false | false |
Example
1 { 2 ... 3 "destination_configuration": { 4 "advanced": { 5 "hard_deletes": false, 6 "table_prefix": "", 7 "truncate_existing": false 8 }, 9 "catalog": "samples", 10 "external_location_name": "", 11 "mirror_schemas": false, 12 "schema": "nyctaxi" 13 } 14 }
Type handling
Destination types
POLYTOMIC TYPE | DATABRICKS TYPE |
---|---|
array<> | ARRAY<> |
bigint | BIGINT |
boolean | BOOLEAN |
date | DATE |
datetime | TIMESTAMP |
decimal(precision, scale) | DECIMAL(precision,scale) |
double | DOUBLE |
int | INT |
json | STRING |
jsonarray | STRING |
number | DECIMAL(38,18) |
object{} | STRUCT<> |
single | FLOAT |
smallint | SMALLINT |
string | STRING |
time | TIMESTAMP |
Source types
DATABRICKS TYPE | POLYTOMIC TYPE |
---|---|
ARRAY<> | array<> |
BIGINT | bigint |
DATE | date |
DECIMAL(precision, scale) | decimal(precision, scale) |
DOUBLE | double |
FLOAT | single |
INT | int |
INTERVAL | string |
MAP<> | object{} |
SMALLINT | smallint |
STRUCT<> | object{} |
TIMESTAMP | datetime |
TIMESTAMP_NTZ | datetime |
TINYINT | smallint |
VARCHAR | string |