AnalysisException: [RequestId=d065497e-b88c-4687-9687-d7180663e7ba ErrorClass=METASTORE_DOES_NOT_EXIST] No metastore assigned for the current workspace
We see the same error, and can trigger it by trying to list a directory with dbutils.
dbutils.fs.ls("some_abfss_path")
-->
[RequestId=74f37535-0483-4822-bd76-83951839b63b ErrorClass=METASTORE_DOES_NOT_EXIST] No metastore assigned for the current workspace.
I did test with different access modes and found that it indeed works with "data_security_mode": "LEGACY_SINGLE_USER_STANDARD". I also found clusters with no entry for "data_security_mode" which worked as expected.