Azure Blob Storage

Supports:

  • ✅ Models
  • ✅ Model sync destination
  • ✅ Bulk sync source
  • ✅ Bulk sync destination

Connection

Configuration

NameTypeDescriptionRequired
account_namestringAccount nametrue
auth_methodstringAuthentication method

Accepted values: access_key , client_credentials , oauth
true
container_namestringContainer nametrue
csv_has_headersbooleanCSV files have headers

Whether CSV files have a header row with field names.
false
is_single_tablebooleanFiles are time-based snapshots

Treat the files as a single table.
false
oauth_refresh_tokenstringfalse

auth_method

When auth_method is access_key
NameTypeDescriptionRequired
access_keystringAccess keytrue
When auth_method is client_credentials
NameTypeDescriptionRequired
client_idstringClient IDtrue
client_secretstringClient secrettrue
tenant_idstringTenant IDtrue

is_single_table

When is_single_table is true
NameTypeDescriptionRequired
is_directory_snapshotbooleanMulti-directory multi-tablefalse
When is_single_table is true AND is_directory_snapshot is false
NameTypeDescriptionRequired
single_table_file_formatstringFile format

Accepted values: csv, json, parquet
false
single_table_namestringCollection namefalse
When is_single_table is true AND is_directory_snapshot is true
NameTypeDescriptionRequired
single_table_file_formatsarrayFile formats that may be present across different tablesfalse

is_directory_snapshot

When is_directory_snapshot is true
NameTypeDescriptionRequired
directory_glob_patternstringTables glob pathtrue

single_table_file_format

When single_table_file_format is csv OR single_table_file_formats contains csv
NameTypeDescriptionRequired
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.
false

Example

{
"name": "Azure Blob Storage connection",
"type": "azureblob",
"configuration": {
"account_name": "account",
"auth_method": "",
"container_name": "container",
"csv_has_headers": false,
"is_single_table": false,
"oauth_refresh_token": "dasfdasz62px8lqeoakuea2ccl4rxm13i6tbyorxhu1i20kc8ruvksmzxq"
}
}

Model Sync

Source

Configuration

NameTypeDescriptionRequired
file_formatstringFile format

Accepted values: csv, json, parquet
false
keystringObject key

The key of the object in the bucket to read from.
false
model_fromstringFiles

The model is generated from a single file or a multi-file archive. Accepted values: single_file, multi_file_archive
true
skip_linesintegerSkip first lines

Skip first N lines of each CSV file.
false
subfolderstringSubfolder to read files from from (optional)false

Example

{
...
"configuration": {
"file_format": "",
"key": "",
"model_from": "",
"skip_lines": 0,
"subfolder": ""
}
}

Target

Azure Blob Storage connections may be used as the destination in a model sync.

All targets

Configuration
NameTypeDescriptionRequired
formatstringOutput format

Output file encoding. Accepted values: csv, json-doc, json, parquet
false
Example
{
...
"target": {
"configuration": {
"format": "csv"
}
}
}

Bulk Sync

Source

Azure Blob Storage connections may be used as a bulk sync source. No additional configuration options are required.

Destination

Configuration

NameTypeDescriptionRequired
advancedobjectfalse
formatstringOutput file encodingfalse
subfolderstringSubfolder to write to (optional)false

Example

{
...
"destination_configuration": {
"advanced": {
"write_incremental": false
},
"format": "csv",
"subfolder": "reports"
}
}