Hi everyone,

I have an image-based deep learning workload running on Azure Databricks, while the training dataset must remain in AWS S3 due to some constraints. We cannot move or replicate the dataset to Azure.
Our current architecture is roughly:
AWS S3 (images) → Unity Catalog Volume → Azure Databricks GPU compute → PyTorch training

The main issue we're seeing is very high file I/O latency and relatively low training throughput. Since the dataset contains a large number of individual image files, reading the images through the UC Volume appears to involve significant network overhead and many individual file reads.
I initially expected Mosaic Streaming / StreamingDataset to improve this because the dataset can be converted into MDS shards and the shards can be downloaded progressively to local storage while training continues.

However, interestingly, in our testing:
Direct image loading from the UC Volume is currently faster than Mosaic Streaming.
I'm trying to understand whether we are missing an important configuration or whether the cross-cloud architecture itself is the primary bottleneck.

A few questions:
1. What is the recommended architecture for training on S3 data from Azure Databricks when the data cannot be moved to Azure?

2. For Mosaic Streaming, what are the recommended values/strategies for:
shard size
num_workers
predownload
cache_limit
shuffle configuration

3. Is there an optimal MDS shard size for image datasets to minimize S3/network overhead?

4. Would increasing DataLoader workers and prefetching significantly improve throughput in this cross-cloud scenario?

5. Would it be better to use local caching of UC Volume files instead of Mosaic Streaming for a multi-epoch image training workload?

6. Are there any Databricks-recommended approaches for measuring whether the bottleneck is S3 → Azure network bandwidth, file-level latency, CPU image decoding, or GPU starvation?

Our main objective is to maximize GPU utilization and training throughput without moving the source dataset out of S3.
Any recommendations, benchmarks, or reference architectures for this type of cross-cloud training setup would be greatly appreciated