@Gen_Vi332 ,

Apologies for the late response,
My experiments with other solutions show that using Volume paths instead of `abfss://` paths can potentially solve the issue.
The drawback is that you need to expose the folder (or its parent) as Volume in Unity Catalog and grant the client access to the volume object. 
`/Volumes/catalog_dev/operational/vlm_source_landing/file_path`
It seems that for many scenarios volume path is treated as local path, which can potentially simplify the code for non spark python operations.