I have an issue when trying to use the command display(dbutils.fs.ls("abfss://test@test.dfs.core.windows.net")). When I execute the command on my personal cluster, it works, and I can see the files. Before that, I set the following configurations:

spark.conf.set("fs.azure.account.auth.type.test.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.test.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.test.dfs.core.windows.net", client_id)
spark.conf.set("fs.azure.account.oauth2.client.secret.test.dfs.core.windows.net", client_secret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.test.dfs.core.windows.net", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token")

However, when I perform the same action on a serverless environment, I get the following error:

Configuration fs.azure.account.auth.type.test.dfs.core.windows.net is not available. SQLSTATE: 42K0I

How can I access files stored in Data Lake with serverless?

Thank you.