Scenario - I have an ADLS g2 account and trying to read a delta table using ABFS driver. I am using Databricks Serverless compute. There are no firewalls in place as I am working with sample data. There's network line of sight between Databricks serverless and ADLS, so network connectivity is not a challenge.
Issue - when I use Databricks Serverless to configure Spark.conf.set properties to authenticate/authorise to ADLS, I encounter the error "[CONFIG_NOT_AVAILABLE] Configuration fs.azure.account.auth.type is not available. SQLSTATE: 42K0I".
I am using code snippet provided here which uses spark.conf.set to set Spark properties. Upon further investigations, turns out that Databricks Serverless, to automate configuration of Spark on serverless compute, does not allow changing configuration.
In absence of this, any suggestions how I can continue using ABFS driver to read a delta table directly.
Note: I am not using UC.
Thanks.
