Reading delta table from ADLS gen2 using ABFS driver

jeet414 — Mon, 10 Mar 2025 01:37:34 GMT

Scenario - I have an ADLS g2 account and trying to read a delta table using ABFS driver. I am using Databricks Serverless compute. There are no firewalls in place as I am working with sample data. There's network line of sight between Databricks serverless and ADLS, so network connectivity is not a challenge.

Issue - when I use Databricks Serverless to configure Spark.conf.set properties to authenticate/authorise to ADLS, I encounter the error "[CONFIG_NOT_AVAILABLE] Configuration fs.azure.account.auth.type is not available. SQLSTATE: 42K0I".

I am using code snippet provided here which uses spark.conf.set to set Spark properties. Upon further investigations, turns out that Databricks Serverless, to automate configuration of Spark on serverless compute, does not allow changing configuration.

In absence of this, any suggestions how I can continue using ABFS driver to read a delta table directly.

Note: I am not using UC.

Thanks.

Re: Reading delta table from ADLS gen2 using ABFS driver

Nivethan_Venkat — Mon, 10 Mar 2025 23:37:13 GMT

Hi @jeet414,

You can always read Delta table from ADLS storage in Databricks mount points: https://learn.microsoft.com/en-us/azure/databricks/dbfs/mounts

This is just experimental, please give it a try.

Best Regards,
Nivethan V

topic Reading delta table from ADLS gen2 using ABFS driver in Administration & Architecture

Reading delta table from ADLS gen2 using ABFS driver

Re: Reading delta table from ADLS gen2 using ABFS driver