topic Databricks Access Issue with UC in Data Engineering

Databricks Access Issue with UC

Jothia — Fri, 21 Nov 2025 07:52:21 GMT

Hi All ,

We are facing issues while reading Storage account where stream data from data verse in Unity catalog through External table but not every time . It was running fine with hive

An error occurred while calling o393.sql.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 84.0 failed 4 times, most recent failure: Lost task 0.3 in stage 84.0 (TID 171) (10.152.188.38 executor 0): com.databricks.sql.io.FileReadException: Error while reading file abfss:REDACTED_LOCAL_PART@storage002.dfs.core.windows.net/account/2019-06.csv.

Any suggestion here ?

Regards,

Jothi.

Re: Databricks Access Issue with UC

Raman_Unifeye — Fri, 21 Nov 2025 08:44:40 GMT

2 Things to check

One:

double-check that you are not trying to authenticate with two different methods (e.g., a cluster credential trying to override the Unity Catalog creds)

The previous Hive setup likely relied on a Cluster-Scoped Service Principal or Shared Access Signature (SAS) key configured directly in the cluster's Spark configuration (e.g., spark.hadoop.fs.azure.account.auth.type). Unity Catalog ignores these cluster-scoped secrets for paths defined in its External Locations. If the table is an External Table managed by Unity Catalog, you must rely on the credentials defined in the External Location.

Two:

Are you not using Autloader?

If the Dataverse stream creates many very small files or is currently in the process of writing/overwriting a file when Spark tries to read it, it can cause transient read failures.

Use Auto Loader if possible, as they handle file discovery and eventual consistency better.

Re: Databricks Access Issue with UC

Jothia — Fri, 21 Nov 2025 08:59:14 GMT

@Raman_Unifeye Thanks your response. We are using external location path under UC only in the External table. Looks no issues with authentication as well we are not getting error always.