This issue appears to be related to Azure Storage access through Unity Catalog rather than the data itself, especially since the same workload was working fine with Hive and the failure is intermittent.

A few areas worth checking:

1. Storage Credential Configuration
Verify that the Storage Credential and External Location used by Unity Catalog are configured correctly.
If you're using a Service Principal, ensure the client secret has not expired and has the required permissions on the storage account.
2. Transient Authentication or Token Refresh Issues
Intermittent failures can sometimes occur due to token refresh or credential caching issues within Unity Catalog.
Check if the failures happen after long-running cluster sessions.
3. Azure Storage Throttling or Network Connectivity
Review Azure Storage metrics for throttling, timeouts, or connectivity-related errors.
Verify whether the issue affects random files or specific files only.
4. File-Level Validation
Since the error references a specific CSV file, confirm that the file is not corrupted and can be accessed directly from the storage account.