cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton
Contributor

Hi Community,

Up until recently I was happily deleting deltaTables in ADLS Gen with their associated _delta_log table, and subsequently recreating the same table with a new _delta_log table.

Now, after deleting a table with its associated _delta_log table when I attempt to create a new table with the same name I get the error:

DeltaIllegalStateException: The protocol of your Delta table could not be recovered while Reconstructing version: 0. Did you manually delete files in the _delta_log directory?

Has Databricks changed something that prevents people from recreating new deltaTables with the same table name?

Can someone please let me know how to resolve this.

 

 

19 REPLIES 19

nayan_wylde
Honored Contributor III

1. If you purposely removed _delta_log/ but kept data files.
Delete any remaining _delta_log/ completely, then convert the Parquet directory back to Delta to create a fresh log:

-- path has only Parquet data now
CONVERT TO DELTA parquet.`abfss://<container>@<account>.dfs.core.windows.net/<path>/`

-------------------------------------------------------------------------------------------------------------------------------------

2. Ensure the path is 100% empty (no hidden _delta_log):

# in a Databricks notebook
display(dbutils.fs.ls("abfss://<container>@<account>.dfs.core.windows.net/<path>"))
display(dbutils.fs.ls("abfss://<container>@<account>.dfs.core.windows.net/<path>/_delta_log"))

If anything remains, remove the entire base folder
dbutils.fs.rm("abfss://<container>@<account>.dfs.core.windows.net/<path>", recurse=True)

Then recreate the table

CREATE OR REPLACE TABLE catalog.schema.table
USING DELTA
LOCATION 'abfss://<container>@<account>.dfs.core.windows.net/<path>'
AS SELECT * FROM some_source;

szymon_dybczak
Esteemed Contributor III

Hi @Carlton ,

This could happen for following reasons:

  • Manually remove underlying files from the Delta log.
  • Run rm commands or other non-Delta operations that remove files from the Delta log.
  • Drop and immediately create a table on top of the same location.

When files are manually removed or not removed correctly, the Delta log versions become non-contiguous. 

You can try to follow this guide to resolve this issue:

Dropping and recreating Delta tables results in a DeltaVersionsNotContiguousException error - Databr...

Also, the way you were doing in (like manually deleting files) is not recommended by Databricks. 
Instead of dropping and recreating Delta tables, the official recommendation is to use CREATE OR REPLACE command.

K_Anudeep
Databricks Employee
Databricks Employee

Hello @Carlton ,

Good Day!

What is the DBR version? How are you creating the table? Did you drop the table and recreate it using SQL?

Carlton
Contributor

Hi Guys,

I really appreciate you getting in touch, this has been driving me crazy.

I delete the entire folder from ADLS. Admittedly, I didn't remove the folder from within Databricks using dbutils.fs.rm. 

If you take a look at my image, I want to remove the 'Country'. So I would delete the Country folder and all sub-folders e.g. folder  and then _delta_log folder. But when attempt to create another Country  deltaTable I get the error.

Are you suggesting that I should delete from within databricks?

K_Anudeep
Databricks Employee
Databricks Employee

Hi @Carlton ,

Yes—delete from inside Databricks (and drop the table) when you plan to reuse the same path/name. External deletes can leave you with a half-cleaned path or stale state, which leads to the “protocol could not be recovered (version: 0)” error.

 

Carlton
Contributor

Sorry I forgot to add the image:

Carlton_0-1758119386566.png

 

 

Carlton
Contributor

Hi nayan_wylde, I'm getting the following error when I try to list the files in account

Carlton_0-1758120870762.png

 

szymon_dybczak
Esteemed Contributor III

Hi,

Delete all folders in ADLS related to a country and also execute DROP TABLE command on the catalog where this table resides.

Probably catalog still stores metadata about this table and thinks it exists.

Then try to create table - but only if you have performed above steps

nayan_wylde
Honored Contributor III

@Carlton This error means you are not passing the client id and client secret in the spark config that will authenticate against the storage or the client secret expired.

Carlton
Contributor

Hi szymon_dybczak, if you take a look at the image, are you suggesting I delete the folder BASE? That folder is the equivalent to our SILVER layer. Deleting that folder would be devastating to our business. I deleted from DATA and all sub-folders below DATA including Country. Shouldn't that be enough?

szymon_dybczak
Esteemed Contributor III

Hi, sorry. I should be more specific. You should delete country folder along with all subfolders ( basically the folder in which the data for the table you want to delete resides).

After that, execute DROP TABLE command on the catalog where the table you wanted to delete was registered.

Carlton
Contributor

Hi szymon_dybczak, no worries, I'm just grateful that you're helping me out. I did what you suggested along with deleting the table from the catalog, see image but I'm still getting the same error

Carlton_0-1758123155442.png

I should mention that I'm getting the following error when I try list files

 

display(dbutils.fs.ls("abfss://root@xxxxxx.dfs.core.windows.net/"))

fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

Carlton
Contributor

I should also mention that not only do I get the error:

DeltaIllegalStateException: The protocol of your Delta table could not be recovered while Reconstructing version: 0. Did you manually delete files in the _delta_log directory?

The table folder that is created looks like the following:

Carlton_1-1758124038734.png

 

 

Carlton
Contributor

I should also mentioned I am using the following code to create the deltaTable which I have been using for years

  deltadf = DeltaTable.forName(spark,f"{stageName}{regName}")

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now