Error running history command twice using DeltaTable

mathiaskvist — Fri, 15 Dec 2023 09:14:51 GMT

I'm using Unity Catalog on Azure with a Managed Identity connected to my Storage Account. I can read and write data without issues, and interact with data using pyspark and SQL.

I can display the history by running a cell with the following code:

from delta.tables import DeltaTable dt = DeltaTable.forName(spark, "CATALOG.SCHEMA.TABLE") dt.history().display()

However, if I run the same history display command but in a new cell I get a strange error:

dt.history().display()

I get the following error:

Failure to initialize configuration for storage account NAME.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

However, I can keep running the first cell multiple times and get a result displayed, but running display in a new cell throws the error.

Also trying to persist the history throws an error under certain circumstances:

from delta.tables import DeltaTable dt = DeltaTable.forName(spark, "CATALOG.SCHEMA.TABLE")

Persist command in a new cell:

a = dt.history().persist()

Throws:

AnalysisException: [RequestId=REQUESTID ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://CONTAINER@STORAGEACCOUNT.dfs.core.windows.net/__unitystorage/catalogs/SOMEID/tables/TABLEID' overlaps with managed storage within 'GenerateTemporaryPathCredential' call

I can run the history command in SQL cell multiple times without any issues:

%sql describe history CATALOG.SCHEMA.TABLE

TLDR: Inconsistent behavior of delta.tables.DeltaTable.history command while sql describe history works

topic Error running history command twice using DeltaTable in Administration & Architecture

Error running history command twice using DeltaTable