We were getting this problem when using directory-scoped SAS tokens. While I know there are a number of potential issues that can cause this problem, one potential explanation is that it turns out there is an undocumented spark setting needed on the cluster to enable directory-scoped SAS tokens. In your cluster's Spark config, add the following:

spark.hadoop.fs.azure.account.hns.enabled true

That solved it for us.