@Morten Stakkelandโ :
Yes, it's possible to configure an autoloader to read from multiple locations.
You can define multiple CloudFiles sources for the autoloader, each pointing to a different container in the same storage account. In your case, since the schemas of the files are identical, you can use the same schema for both sources. Here's an example of how you can define multiple sources in your autoloader configuration:
{
"format": "delta",
"mode": "append",
"cloudFiles": {
"cloudStorage": {
"timeout": "1h",
"accountName": "<storage-account-name>",
"accountKey": "<storage-account-access-key>"
},
"useIncrementalListing": true,
"maxConcurrentFileCount": 20,
"source": [
{
"path": "/container1/",
"globPattern": "*.csv",
"recursive": true
},
{
"path": "/container2/",
"globPattern": "*.csv",
"recursive": true
}
]
}
}
In this example, we define two sources, one for the /container1/ directory and one for the /container2/ directory. The globPattern parameter specifies that we only want to load CSV files, and the recursive parameter tells the autoloader to recursively search for files in subdirectories.
Note that you can also use different schemas for the two sources if necessary, as long as they have the same column names and data types.
