Databricks

Supports:

✅ Models
✅ Model sync destination
✅ Bulk sync source
✅ Bulk sync destination

Connection

Configuration

Name	Type	Description	Required
`cloud_provider`	`string`	Cloud Provider (destination support only) Accepted values: `aws` ↓, `azure` ↓	`false`
`databricks_auth_mode`	`string`	Authentication Method Accepted values: `access_token` ↓, `oauth_service_principal` ↓	`true`
`enable_delta_uniform`	`boolean`	Enable Delta UniForm tables	`false`
`enforce_query_limit`	`boolean`	Limit concurrent queries ↓	`false`
`http_path`	`string`	HTTP Path	`true`
`port`	`integer`	Port	`true`
`server_hostname`	`string`	Server Hostname	`true`
`ssh`	`boolean`	Connect over SSH tunnel ↓	`false`
`unity_catalog_enabled`	`boolean`	Unity Catalog enabled	`false`
`use_bulk_sync_staging_schema`	`boolean`	Use custom bulk sync staging schema ↓	`false`

`databricks_auth_mode`

When `databricks_auth_mode` is `access_token`

Name	Type	Description	Required
`access_token`	`string`	Access Token	`true`

When `databricks_auth_mode` is `oauth_service_principal`

Name	Type	Description	Required
`service_principal_id`	`string`	Service Principal ID	`true`
`service_principal_secret`	`string`	Service Principal Secret	`true`

`cloud_provider`

When `cloud_provider` is `aws`

Name	Type	Description	Required
`auth_mode`	`string`	AWS Authentication Method How to authenticate with AWS. Defaults to Access Key and Secret. Accepted values: `access_key_and_secret`, `iam_role`	`true`
`s3_bucket_name`	`string`	S3 Bucket Name (destinations only) Name of bucket used for staging data load files	`false`
`s3_bucket_region`	`string`	S3 Bucket Region (destinations only) Region of bucket	`false`
`set_retention_properties`	`boolean`	Configure data retention for tables	`false`

When `cloud_provider` is `azure`

Name	Type	Description	Required
`azure_access_key`	`string`	Storage Account Access Key (destination support only) The access key associated with this storage account	`false`
`azure_account_name`	`string`	Storage Account Name (destination support only) The account name of the storage account	`false`
`container_name`	`string`	Storage Container Name (destination support only) The container which we will stage files in	`false`

`use_bulk_sync_staging_schema`

When `use_bulk_sync_staging_schema` is `true`

Name	Type	Description	Required
`bulk_sync_staging_schema`	`string`	Staging schema name	`false`

`enforce_query_limit`

When `enforce_query_limit` is `true`

Name	Type	Description	Required
`concurrent_queries`	`integer`	Concurrent query limit	`false`

`ssh`

When `ssh` is `true`

Name	Type	Description	Required
`ssh_blob_storage`	`boolean`	Use SSH for cloud storage bucket	`false`
`ssh_host`	`string`	SSH host	`false`
`ssh_port`	`integer`	SSH port	`false`
`ssh_private_key`	`string`	Private key	`false`
`ssh_user`	`string`	SSH user	`false`

`set_retention_properties`

When `set_retention_properties` is `true`

Name	Type	Description	Required
`deleted_file_retention_days`	`integer`	Deleted file retention	`false`
`log_file_retention_days`	`integer`	Log retention	`false`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

Name	Type	Description	Required
`aws_access_key_id`	`string`	AWS Access Key ID (destinations only) See https://docs.polytomic.com/docs/databricks-connections#writing-to-databricks	`true`
`aws_secret_access_key`	`string`	AWS Secret Access Key (destinations only)	`true`

When `auth_mode` is `iam_role`

Name	Type	Description	Required
`iam_role_arn`	`string`	IAM Role ARN	`true`
`storage_credential_name`	`string`	Storage credential name	`false`

Example

1 {
2   "name": "Databricks connection",
3   "type": "databricks",
4   "configuration": {
5     "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
6     "auth_mode": "access_key_and_secret",
7     "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
8     "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
9     "cloud_provider": "aws",
10     "databricks_auth_mode": "access_token",
11     "enable_delta_uniform": false,
12     "enforce_query_limit": false,
13     "http_path": "/sql",
14     "port": 443,
15     "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
16     "s3_bucket_region": "us-east-1",
17     "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
18     "set_retention_properties": false,
19     "ssh": false,
20     "unity_catalog_enabled": false,
21     "use_bulk_sync_staging_schema": false
22   }
23 }

Read-only properties

Name	Type	Description	Required

`auth_mode`

When `auth_mode` is `access_key_and_secret`

Name	Type	Description	Required
`aws_user`	`string`	User ARN (destinations only)	`false`

When `auth_mode` is `iam_role`

Name	Type	Description	Required
`external_id`	`string`	External ID for the IAM role	`false`

Model Sync

Source

Configuration

Name	Type	Description	Required
`catalog`	`string`	Catalog	`false`
`query`	`string`	Query	`false`
`schema`	`string`	Schema	`false`
`table`	`string`	Table	`false`

Example

1 {
2   ...
3   "configuration": {
4     "catalog": "samples",
5     "query": "SELECT * FROM samples.nyctaxi.trips",
6     "schema": "nyctaxi",
7     "table": "trips"
8   }
9 }

Target

Databricks connections may be used as the destination in a model sync.

All targets

Configuration

Name	Type	Description	Required
`created_column`	`string`	’Created at’ timestamp column	`false`
`preserve_table_on_resync`	`boolean`	Preserve destination table when resyncing	`false`
`updated_column`	`string`	’Updated at’ timestamp column	`false`
`write_record_timestamps`	`boolean`	Write row timestamp metadata	`false`

Example

1 {
2   ...
3   "target": {
4       "configuration": {
5         "created_column": "",
6         "preserve_table_on_resync": false,
7         "updated_column": "",
8         "write_record_timestamps": false
9     }
10   }
11 }

Target creation

Databricks connections may be used to create a new target for a model sync. The following parameters are required to create a new target:

NAME	DESCRIPTION	ENUM
name	Table name	false

Bulk Sync

Source

Databricks connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

Name	Type	Description	Required
`advanced`	`object`		`false`
`catalog`	`string`	Catalog	`false`
`external_location_name`	`string`	External location	`false`
`mirror_schemas`	`boolean`	Mirror schemas	`false`
`schema`	`string`	Output schema	`false`

Example

1 {
2   ...
3   "destination_configuration": {
4     "advanced": {
5         "deleted_file_retention_days": 0,
6         "empty_strings_null": false,
7         "enable_change_data_feed": false,
8         "hard_deletes": false,
9         "log_file_retention_days": 0,
10         "set_retention_properties": false,
11         "table_prefix": "",
12         "truncate_existing": false
13     },
14     "catalog": "samples",
15     "external_location_name": "",
16     "mirror_schemas": false,
17     "schema": "nyctaxi"
18   }
19 }

Type handling

Destination types

POLYTOMIC TYPE	DATABRICKS TYPE
`array<>`	`ARRAY<>`
`bigint`	`BIGINT`
`boolean`	`BOOLEAN`
`date`	`DATE`
`datetime`	`TIMESTAMP`
`decimal(precision, scale)`	`DECIMAL(precision,scale)`
`double`	`DOUBLE`
`int`	`INT`
`json`	`STRING`
`jsonarray`	`STRING`
`number`	`DECIMAL(38,18)`
`object{}`	`STRING`
`single`	`FLOAT`
`smallint`	`SMALLINT`
`string`	`STRING`
`time`	`TIMESTAMP`

Source types

DATABRICKS TYPE	POLYTOMIC TYPE
`ARRAY<>`	`array<>`
`BIGINT`	`bigint`
`DATE`	`date`
`DECIMAL(precision, scale)`	`decimal(precision, scale)`
`DOUBLE`	`double`
`FLOAT`	`single`
`INT`	`int`
`INTERVAL`	`string`
`MAP<>`	`object{}`
`SMALLINT`	`smallint`
`STRUCT<>`	`object{}`
`TIMESTAMP`	`datetime_tz`
`TIMESTAMP_NTZ`	`datetime`
`TINYINT`	`smallint`
`VARCHAR`	`string`

1	{
2	"name": "Databricks connection",
3	"type": "databricks",
4	"configuration": {
5	"access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
6	"auth_mode": "access_key_and_secret",
7	"aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
8	"aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
9	"cloud_provider": "aws",
10	"databricks_auth_mode": "access_token",
11	"enable_delta_uniform": false,
12	"enforce_query_limit": false,
13	"http_path": "/sql",
14	"port": 443,
15	"s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
16	"s3_bucket_region": "us-east-1",
17	"server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
18	"set_retention_properties": false,
19	"ssh": false,
20	"unity_catalog_enabled": false,
21	"use_bulk_sync_staging_schema": false
22	}
23	}

1	{
2	...
3	"configuration": {
4	"catalog": "samples",
5	"query": "SELECT * FROM samples.nyctaxi.trips",
6	"schema": "nyctaxi",
7	"table": "trips"
8	}
9	}

1	{
2	...
3	"target": {
4	"configuration": {
5	"created_column": "",
6	"preserve_table_on_resync": false,
7	"updated_column": "",
8	"write_record_timestamps": false
9	}
10	}
11	}

1	{
2	...
3	"destination_configuration": {
4	"advanced": {
5	"deleted_file_retention_days": 0,
6	"empty_strings_null": false,
7	"enable_change_data_feed": false,
8	"hard_deletes": false,
9	"log_file_retention_days": 0,
10	"set_retention_properties": false,
11	"table_prefix": "",
12	"truncate_existing": false
13	},
14	"catalog": "samples",
15	"external_location_name": "",
16	"mirror_schemas": false,
17	"schema": "nyctaxi"
18	}
19	}

Connection

Configuration

databricks_auth_mode

When databricks_auth_mode is access_token

When databricks_auth_mode is oauth_service_principal

cloud_provider

When cloud_provider is aws

When cloud_provider is azure

use_bulk_sync_staging_schema

When use_bulk_sync_staging_schema is true

enforce_query_limit

When enforce_query_limit is true

ssh

When ssh is true

set_retention_properties

When set_retention_properties is true

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

Example

Read-only properties

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

Model Sync

Source

Configuration

Example

Target

All targets

Configuration

Example

Target creation

Bulk Sync

Source

Destination

Configuration

Example

Type handling

Destination types

Source types

Connection

Configuration

databricks_auth_mode

When databricks_auth_mode is access_token

When databricks_auth_mode is oauth_service_principal

cloud_provider

When cloud_provider is aws

When cloud_provider is azure

use_bulk_sync_staging_schema

When use_bulk_sync_staging_schema is true

enforce_query_limit

When enforce_query_limit is true

ssh

When ssh is true

set_retention_properties

When set_retention_properties is true

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

Example

Read-only properties

auth_mode

When auth_mode is access_key_and_secret

When auth_mode is iam_role

Model Sync

Source

Configuration

Example

Target

All targets

Configuration

Example

Target creation

Bulk Sync

Source

Destination

Configuration

Example

Type handling

`databricks_auth_mode`

When `databricks_auth_mode` is `access_token`

When `databricks_auth_mode` is `oauth_service_principal`

`cloud_provider`

When `cloud_provider` is `aws`

When `cloud_provider` is `azure`

`use_bulk_sync_staging_schema`

When `use_bulk_sync_staging_schema` is `true`

`enforce_query_limit`

When `enforce_query_limit` is `true`

`ssh`

When `ssh` is `true`

`set_retention_properties`

When `set_retention_properties` is `true`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`

`databricks_auth_mode`

When `databricks_auth_mode` is `access_token`

When `databricks_auth_mode` is `oauth_service_principal`

`cloud_provider`

When `cloud_provider` is `aws`

When `cloud_provider` is `azure`

`use_bulk_sync_staging_schema`

When `use_bulk_sync_staging_schema` is `true`

`enforce_query_limit`

When `enforce_query_limit` is `true`

`ssh`

When `ssh` is `true`

`set_retention_properties`

When `set_retention_properties` is `true`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`

`auth_mode`

When `auth_mode` is `access_key_and_secret`

When `auth_mode` is `iam_role`