Skip to navigation
Configuring your connectionsConnection configurations

Databricks

Supports:

  • ✅ Models
  • ✅ Model sync destination
  • ✅ Bulk sync source
  • ✅ Bulk sync destination

Connection

Configuration

NameTypeDescriptionRequired
cloud_providerstringCloud provider (destination support only)

Accepted values: aws ↓, azure ↓		optional
databricks_auth_modestringAuthentication method

Accepted values: access_token ↓, oauth_service_principal ↓		required
enable_delta_uniformbooleanEnable Delta UniForm tablesoptional
enforce_query_limitbooleanLimit concurrent queries ↓optional
http_pathstringHTTP pathrequired
portintegerPortrequired
server_hostnamestringServer hostnamerequired
sshbooleanConnect over SSH tunnel ↓optional
unity_catalog_enabledbooleanUnity Catalog enabled ↓optional
use_bulk_sync_staging_schemabooleanUse custom bulk sync staging schema ↓optional

cloud_provider = aws

NameTypeDescriptionRequired
auth_modestringAWS authentication method

How to authenticate with AWS. Defaults to Access Key and Secret. Accepted values: access_key_and_secret ↓, iam_role ↓		required
s3_bucket_namestringS3 bucket name (destinations only)

Name of bucket used for staging data load files		optional
s3_bucket_regionstringS3 bucket region (destinations only)

Region of bucket		optional
set_retention_propertiesbooleanConfigure data retention for tables ↓optional
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "cloud_provider": "aws",
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": false,
    "ssh": false,
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}
auth_mode

When auth_mode is access_key_and_secret:

NameTypeDescriptionRequired
aws_access_key_idstringAWS access key ID (destinations only)

See https://docs.polytomic.com/docs/databricks-connections#writing-to-databricks		required
aws_secret_access_keystringAWS secret access key (destinations only)required
Read-only propertyTypeDescription
aws_userstringUser ARN (destinations only)

When auth_mode is iam_role:

NameTypeDescriptionRequired
iam_role_arnstringIAM role ARNrequired
storage_credential_namestringStorage credential nameoptional
Read-only propertyTypeDescription
external_idstringExternal ID for the IAM role
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "iam_role",
    "cloud_provider": "aws",
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "iam_role_arn": "",
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": false,
    "ssh": false,
    "storage_credential_name": "",
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}
set_retention_properties

When set_retention_properties is true:

NameTypeDescriptionRequired
deleted_file_retention_daysintegerDeleted file retentionoptional
log_file_retention_daysintegerLog retentionoptional
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "cloud_provider": "aws",
    "databricks_auth_mode": "access_token",
    "deleted_file_retention_days": 0,
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "log_file_retention_days": 0,
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": true,
    "ssh": false,
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}

cloud_provider = azure

NameTypeDescriptionRequired
azure_access_keystringStorage account access key (destination support only)

The access key associated with this storage account		optional
azure_account_namestringStorage account name (destination support only)

The account name of the storage account		optional
container_namestringStorage container name (destination support only)

The container which we will stage files in		optional
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "azure_access_key": "abcdefghijklmnopqrstuvwxyz0123456789/+ABCDEabcdefghijklmnopqrstuvwxyz0123456789/+ABCDE==",
    "azure_account_name": "account",
    "cloud_provider": "azure",
    "container_name": "container",
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "port": 443,
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "ssh": false,
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}

databricks_auth_mode

When databricks_auth_mode is access_token:

NameTypeDescriptionRequired
access_tokenstringAccess tokenrequired

When databricks_auth_mode is oauth_service_principal:

NameTypeDescriptionRequired
service_principal_idstringService principal IDrequired
service_principal_secretstringService principal secretrequired
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "cloud_provider": "aws",
    "databricks_auth_mode": "oauth_service_principal",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "service_principal_id": "sp-1234abcd",
    "service_principal_secret": "abcdefghijklmnopqrstuvwxyz0123456789/+ABCDEabcdefghijklmnopqrstuvwxyz0123456789/+ABCDE==",
    "set_retention_properties": false,
    "ssh": false,
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}

enforce_query_limit

When enforce_query_limit is true:

NameTypeDescriptionRequired
concurrent_queriesintegerConcurrent query limitoptional
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "cloud_provider": "aws",
    "concurrent_queries": 0,
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": true,
    "http_path": "/sql",
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": false,
    "ssh": false,
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}

ssh

When ssh is true:

NameTypeDescriptionRequired
ssh_blob_storagebooleanUse SSH for cloud storage bucketoptional
ssh_hoststringSSH hostrequired
ssh_portintegerSSH portrequired
ssh_private_keystringPrivate keyrequired
ssh_userstringSSH userrequired
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "cloud_provider": "aws",
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": false,
    "ssh": true,
    "ssh_blob_storage": false,
    "ssh_host": "bastion.example.com",
    "ssh_port": 22,
    "ssh_private_key": "",
    "ssh_user": "root",
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": false
  }
}

unity_catalog_enabled

When unity_catalog_enabled is true:

NameTypeDescriptionRequired
include_hive_metastorebooleanInclude legacy Hive metastore tablesoptional
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "cloud_provider": "aws",
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "include_hive_metastore": false,
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": false,
    "ssh": false,
    "unity_catalog_enabled": true,
    "use_bulk_sync_staging_schema": false
  }
}

use_bulk_sync_staging_schema

When use_bulk_sync_staging_schema is true:

NameTypeDescriptionRequired
bulk_sync_staging_schemastringStaging schema namerequired
{
  "name": "Databricks connection",
  "type": "databricks",
  "configuration": {
    "access_token": "isoz8af6zvp8067gu68gvrp0oftevn",
    "auth_mode": "access_key_and_secret",
    "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
    "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "bulk_sync_staging_schema": "",
    "cloud_provider": "aws",
    "databricks_auth_mode": "access_token",
    "enable_delta_uniform": false,
    "enforce_query_limit": false,
    "http_path": "/sql",
    "port": 443,
    "s3_bucket_name": "s3://polytomic-databricks-results/customer-dataset",
    "s3_bucket_region": "us-east-1",
    "server_hostname": "dbc-1234dsafas-d0001.cloud.databricks.com",
    "set_retention_properties": false,
    "ssh": false,
    "unity_catalog_enabled": false,
    "use_bulk_sync_staging_schema": true
  }
}

Model Sync

Source

Configuration

NameTypeDescriptionRequired
catalogstringCatalogoptional
querystringQueryoptional
schemastringSchemaoptional
tablestringTableoptional

Example

{
  ...
  "configuration": {
    "catalog": "samples",
    "query": "SELECT * FROM samples.nyctaxi.trips",
    "schema": "nyctaxi",
    "table": "trips"
  }
}

Target

Databricks connections may be used as the destination in a model sync.

All targets

Configuration
NameTypeDescriptionRequired
created_columnstring’Created at’ timestamp columnoptional
preserve_table_on_resyncbooleanPreserve destination table when resyncingoptional
updated_columnstring’Updated at’ timestamp columnoptional
write_record_timestampsbooleanWrite row timestamp metadataoptional
Example
{
  ...
  "target": {
      "configuration": {
        "created_column": "",
        "preserve_table_on_resync": false,
        "updated_column": "",
        "write_record_timestamps": false
    }
  }
}

Target creation

Databricks connections may be used to create a new target for a model sync. The following parameters are required to create a new target:

NAMEDESCRIPTIONENUM
nameTable namefalse

Bulk Sync

Source

Databricks connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

NameTypeDescriptionRequired
advancedobjectoptional
catalogstringCatalogoptional
external_location_namestringExternal locationoptional
mirror_schemasbooleanMirror schemasoptional
schemastringOutput schemaoptional

Example

{
  ...
  "destination_configuration": {
    "advanced": {
        "deleted_file_retention_days": 0,
        "empty_strings_null": false,
        "enable_change_data_feed": false,
        "hard_deletes": false,
        "initial_execution": "rebuild",
        "log_file_retention_days": 0,
        "set_retention_properties": false,
        "table_prefix": "",
        "truncate_existing": false
    },
    "catalog": "samples",
    "external_location_name": "",
    "mirror_schemas": false,
    "schema": "nyctaxi"
  }
}

Type handling

Destination types

POLYTOMIC TYPEDATABRICKS TYPE
array<>ARRAY<>
bigintBIGINT
booleanBOOLEAN
dateDATE
datetimeTIMESTAMP
decimal(precision, scale)DECIMAL(precision,scale)
doubleDOUBLE
intINT
jsonSTRING
jsonarraySTRING
numberDECIMAL(38,18)
object{}STRING
singleFLOAT
smallintSMALLINT
stringSTRING
timeTIMESTAMP

Source types

DATABRICKS TYPEPOLYTOMIC TYPE
ARRAY<>array<>
BIGINTbigint
BYTEsmallint
DATEdate
DECIMAL(precision, scale)decimal(precision, scale)
DOUBLEdouble
FLOATsingle
INTint
INTERVALstring
LONGbigint
MAP<>object{}
SHORTsmallint
SMALLINTsmallint
STRUCT<>object{}
TIMESTAMPdatetime_tz
TIMESTAMP_NTZdatetime
TINYINTsmallint
VARCHARstring