Google Cloud Storage

Supports:

  • ✅ Models
  • ✅ Model sync destination
  • ✅ Bulk sync source
  • ✅ Bulk sync destination

Connection

Configuration

NameTypeDescriptionRequired
bucketstringBuckettrue
csv_has_headersbooleanCSV files have headers

Whether CSV files have a header row with field names.
false
is_single_tablebooleanFiles are time-based snapshots

Treat the files as a single table.
false
service_accountstringService account keytrue

is_single_table

When is_single_table is true
NameTypeDescriptionRequired
is_directory_snapshotbooleanMulti-directory multi-tablefalse
When is_single_table is true AND is_directory_snapshot is false
NameTypeDescriptionRequired
single_table_file_formatstringFile format

Accepted values: csv, json, parquet
false
single_table_namestringCollection namefalse
When is_single_table is true AND is_directory_snapshot is true
NameTypeDescriptionRequired
single_table_file_formatsarrayFile formats that may be present across different tablesfalse

is_directory_snapshot

When is_directory_snapshot is true
NameTypeDescriptionRequired
directory_glob_patternstringTables glob pathtrue

single_table_file_format

When single_table_file_format is csv OR single_table_file_formats contains csv
NameTypeDescriptionRequired
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.
false

Example

1{
2 "name": "Google Cloud Storage connection",
3 "type": "gcs",
4 "configuration": {
5 "bucket": "my-bucket",
6 "csv_has_headers": false,
7 "is_single_table": false,
8 "service_account": ""
9 }
10}

Read-only properties

NameTypeDescriptionRequired
client_emailstringService account identityfalse
project_idstringService account project IDfalse

Model Sync

Source

Configuration

NameTypeDescriptionRequired
file_formatstringFile format

Accepted values: csv, json, parquet
false
keystringObject key

The key of the object in the bucket to read from.
false
model_fromstringFiles

The model is generated from a single file or a multi-file archive. Accepted values: single_file, multi_file_archive
true
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.
false
subfolderstringSubfolder to read files from from (optional)false

Example

1{
2 ...
3 "configuration": {
4 "file_format": "",
5 "key": "",
6 "model_from": "",
7 "skip_lines": 0,
8 "subfolder": ""
9 }
10}

Target

Google Cloud Storage connections may be used as the destination in a model sync.

All targets

Configuration
NameTypeDescriptionRequired
formatstringOutput format

Output file encoding. Accepted values: csv, json-doc, json, parquet
false
Example
1{
2 ...
3 "target": {
4 "configuration": {
5 "format": "csv"
6 }
7 }
8}

Bulk Sync

Destination

Configuration

NameTypeDescriptionRequired
formatstringOutput file encodingfalse
subfolderstringSubfolder to write to (optional)false

Example

1{
2 ...
3 "destination_configuration": {
4 "format": "csv",
5 "subfolder": "reports"
6 }
7}