Google BigQuery | Polytomic Docs

Supports:

✅ Models
✅ Model sync destination
✅ Bulk sync source
✅ Bulk sync destination

Connection

Configuration

Name	Type	Description	Required
`auth_method`	`string`	Authentication method Accepted values: `service_account_key` ↓, `workload_identity_federation` ↓, `application_default_credentials`	`true`
`location`	`string`	Location (optional) Region or multi-region for query operations	`false`
`structured_values_as_json`	`boolean`	Write object and array values as JSON	`false`
`use_bulk_sync_staging_schema`	`boolean`	Use custom bulk sync staging schema ↓	`false`
`use_extract`	`boolean`	Use extract for bulk sync from BigQuery ↓	`false`

`use_extract`

When `use_extract` is `true`

Name	Type	Description	Required
`bucket`	`string`	Google Cloud Storage bucket	`true`

`use_bulk_sync_staging_schema`

When `use_bulk_sync_staging_schema` is `true`

Name	Type	Description	Required
`bulk_sync_staging_schema`	`string`	Staging schema name	`true`

`auth_method`

When `auth_method` is `workload_identity_federation`

Name	Type	Description	Required
`credential_config`	`string`	Credential configuration JSON file downloaded from Google Cloud	`true`
`wif_project_id`	`string`	Google Cloud project ID	`true`

When `auth_method` is `service_account_key|application_default_credentials`

Name	Type	Description	Required
`override_project_id`	`string`	Override project ID (optional) Override the default project ID for cross-project access	`false`

When `auth_method` is `service_account_key`

Name	Type	Description	Required
`service_account`	`string`	Service account key	`true`

Example

{
  "name": "Google BigQuery connection",
  "type": "bigquery",
  "configuration": {
    "auth_method": "",
    "location": "us-east1",
    "structured_values_as_json": false,
    "use_bulk_sync_staging_schema": false,
    "use_extract": false
  }
}

Read-only properties

Name	Type	Description	Required

`auth_method`

When `auth_method` is `service_account_key`

Name	Type	Description	Required
`client_email`	`string`	Service account identity	`false`
`project_id`	`string`	Service account project ID	`false`

Model Sync

Source

Configuration

Name	Type	Description	Required
`dataset`	`string`	Dataset	`false`
`query`	`string`		`false`
`table`	`string`	Table	`false`

Example

{
  ...
  "configuration": {
    "dataset": "nyctaxi",
    "query": "SELECT * from sampledata.nyctaxi.taxis",
    "table": "taxis"
  }
}

Target

Google BigQuery connections may be used as the destination in a model sync.

Target creation

Google BigQuery connections may be used to create a new target for a model sync. The following parameters are required to create a new target:

NAME	DESCRIPTION	ENUM
dataset	Dataset	true
name	Table name	false

Bulk Sync

Source

Google BigQuery connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

Name	Type	Description	Required
`advanced`	`object`		`false`
`dataset`	`string`	Output dataset	`false`
`mirror_schemas`	`boolean`	Mirror schemas	`false`

Example

{
  ...
  "destination_configuration": {
    "advanced": {
        "empty_strings_null": false,
        "expire_tables_after_inactivity": false,
        "hard_deletes": false,
        "initial_execution": "",
        "table_expiry_days": 0,
        "table_prefix": "",
        "truncate_existing": false
    },
    "dataset": "dataset",
    "mirror_schemas": false
  }
}

Type handling

Destination types

POLYTOMIC TYPE	GOOGLE BIGQUERY TYPE
`array<>`	`JSON`
`bigint`	`INTEGER`
`boolean`	`BOOLEAN`
`date`	`DATE`
`datetime`	`TIMESTAMP`
`decimal(precision, scale)`	`BIGNUMERIC(precision,scale)`
`double`	`FLOAT`
`int`	`INTEGER`
`json`	`JSON`
`jsonarray`	`JSON`
`number`	`BIGNUMERIC`
`object{}`	`JSON`
`single`	`FLOAT`
`smallint`	`INTEGER`
`string`	`STRING`
`time`	`TIME`

Source types

GOOGLE BIGQUERY TYPE	POLYTOMIC TYPE
`ARRAY<>`	`array<>`
`BIGNUMERIC`	`number`
`BIGNUMERIC(precision, scale)`	`decimal(precision, scale)`
`DATE`	`date`
`FLOAT`	`double`
`INTEGER`	`bigint`
`NUMERIC`	`number`
`NUMERIC(precision, scale)`	`decimal(precision, scale)`
`RECORD`	`json`
`RECORD<>`	`object{}`
`TIME`	`time`
`TIMESTAMP`	`datetime`

Connection

Configuration

use_extract

When use_extract is true

use_bulk_sync_staging_schema

When use_bulk_sync_staging_schema is true

auth_method

When auth_method is workload_identity_federation

When auth_method is service_account_key|application_default_credentials

When auth_method is service_account_key

Example

Read-only properties

auth_method

When auth_method is service_account_key

Model Sync

Source

Configuration

Example

Target

Target creation

Bulk Sync

Source

Destination

Configuration

Example

Type handling

Destination types

Source types

`use_extract`

When `use_extract` is `true`

`use_bulk_sync_staging_schema`

When `use_bulk_sync_staging_schema` is `true`

`auth_method`

When `auth_method` is `workload_identity_federation`

When `auth_method` is `service_account_key|application_default_credentials`

When `auth_method` is `service_account_key`

`auth_method`

When `auth_method` is `service_account_key`