Google Cloud Storage

Supports:

  • ✅ Models
  • ✅ Model sync destination
  • ✅ Bulk sync source
  • ✅ Bulk sync destination

Connection

Configuration

NameTypeDescriptionRequired
bucketstringBuckettrue
csv_has_headersbooleanCSV files have headers

Whether CSV files have a header row with field names.
false
is_single_tablebooleanFiles are time-based snapshots

Treat the files as a single table.
false
service_accountstringService account keytrue

is_single_table

When is_single_table is true
NameTypeDescriptionRequired
is_directory_snapshotbooleanMulti-directory multi-tablefalse
When is_single_table is true AND is_directory_snapshot is false
NameTypeDescriptionRequired
single_table_file_formatstringFile format

Accepted values: csv, json, parquet
false
single_table_namestringCollection namefalse
When is_single_table is true AND is_directory_snapshot is true
NameTypeDescriptionRequired
single_table_file_formatsarrayFile formats that may be present across different tablesfalse

is_directory_snapshot

When is_directory_snapshot is true
NameTypeDescriptionRequired
directory_glob_patternstringTables glob pathtrue

single_table_file_format

When single_table_file_format is csv OR single_table_file_formats contains csv
NameTypeDescriptionRequired
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.
false

Example

{
"name": "Google Cloud Storage connection",
"type": "gcs",
"configuration": {
"bucket": "my-bucket",
"csv_has_headers": false,
"is_single_table": false,
"service_account": ""
}
}

Read-only properties

NameTypeDescriptionRequired
client_emailstringService account identityfalse
project_idstringService account project IDfalse

Model Sync

Source

Configuration

NameTypeDescriptionRequired
file_formatstringFile format

Accepted values: csv, json, parquet
false
keystringObject key

The key of the object in the bucket to read from.
false
model_fromstringFiles

The model is generated from a single file or a multi-file archive. Accepted values: single_file, multi_file_archive
true
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.
false
subfolderstringSubfolder to read files from from (optional)false

Example

{
...
"configuration": {
"file_format": "",
"key": "",
"model_from": "",
"skip_lines": 0,
"subfolder": ""
}
}

Target

Google Cloud Storage connections may be used as the destination in a model sync.

All targets

Configuration
NameTypeDescriptionRequired
formatstringOutput format

Output file encoding. Accepted values: csv, json-doc, json, parquet
false
Example
{
...
"target": {
"configuration": {
"format": "csv"
}
}
}

Bulk Sync

Source

Google Cloud Storage connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

NameTypeDescriptionRequired
advancedobjectfalse
formatstringOutput file encodingfalse
subfolderstringSubfolder to write to (optional)false

Example

{
...
"destination_configuration": {
"advanced": {
"write_incremental": false
},
"format": "csv",
"subfolder": "reports"
}
}