Google Cloud Storage

Supports:

  • ✅ Models
  • ✅ Model sync destination
  • ✅ Bulk sync source
  • ✅ Bulk sync destination

Connection

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
project_idstringfalsefalse
client_emailstringfalsefalse
service_accountstringtruefalse
bucketstringtruefalse
is_single_tablebooleanTreat the files as a single table.falsefalse
is_directory_snapshotbooleanfalsefalse
directory_glob_patternstringfalsefalse
single_table_namestringfalsefalse
single_table_file_formatstringAccepted Values: csv, json, parquetfalsefalse
skip_linesintegerSkip first N lines of each CSV file.falsefalse

Example

1{
2 "name": "Google Cloud Storage connection",
3 "type": "gcs",
4 "configuration": {
5 "bucket": "my-bucket",
6 "client_email": "",
7 "directory_glob_pattern": "",
8 "is_directory_snapshot": false,
9 "is_single_table": false,
10 "project_id": "",
11 "service_account": "",
12 "single_table_file_format": "csv",
13 "single_table_name": "collection",
14 "skip_lines": 0
15 }
16}

Model Sync

Source

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
model_fromstringThe model is generated from a single file or a multi-file archive. Accepted Values: single_file, multi_file_archivetruefalse
file_formatstringFile format Accepted Values: csv, json, parquetfalsefalse
skip_linesintegerSkip first N lines of each CSV file.falsefalse
keystringThe key of the object in the bucket to read from.falsefalse
subfolderstringSubfolder to read files from from (optional)falsefalse

Example

1{
2 ...
3 "configuration": {
4 "file_format": "",
5 "key": "",
6 "model_from": "",
7 "skip_lines": 0,
8 "subfolder": ""
9 }
10}

Target

Google Cloud Storage connections may be used as the destination in a model sync.

All targets

Configuration
NAMETYPEDESCRIPTIONREQUIREDREADONLY
formatstringOutput file encodingfalsefalse
Example
1{
2 ...
3 "target": {
4 "configuration": {
5 "format": "csv"
6 }
7 }
8}

Bulk Sync

Destination

Configuration

NAMETYPEDESCRIPTIONREQUIREDREADONLY
subfolderstringSubfolder to write to (optional)falsefalse
formatstringOutput file encodingfalsefalse

Example

1{
2 ...
3 "destination_configuration": {
4 "format": "csv",
5 "subfolder": "reports"
6 }
7}