topic Re: No Longer Able to Create DeltaTables in ADLS Gen 2 in Data Engineering

No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 13:45:11 GMT

Hi Community,

Up until recently I was happily deleting deltaTables in ADLS Gen with their associated _delta_log table, and subsequently recreating the same table with a new _delta_log table.

Now, after deleting a table with its associated _delta_log table when I attempt to create a new table with the same name I get the error:

DeltaIllegalStateException: The protocol of your Delta table could not be recovered while Reconstructing version: 0. Did you manually delete files in the _delta_log directory?

Has Databricks changed something that prevents people from recreating new deltaTables with the same table name?

Can someone please let me know how to resolve this.

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

nayan_wylde — Wed, 17 Sep 2025 14:06:36 GMT

1. If you purposely removed _delta_log/ but kept data files.
Delete any remaining _delta_log/ completely, then convert the Parquet directory back to Delta to create a fresh log:

-- path has only Parquet data now
CONVERT TO DELTA parquet.`abfss://<container>@<account>.dfs.core.windows.net/<path>/`

-------------------------------------------------------------------------------------------------------------------------------------

2. Ensure the path is 100% empty (no hidden _delta_log):

# in a Databricks notebook
display(dbutils.fs.ls("abfss://<container>@<account>.dfs.core.windows.net/<path>"))
display(dbutils.fs.ls("abfss://<container>@<account>.dfs.core.windows.net/<path>/_delta_log"))

If anything remains, remove the entire base folder
dbutils.fs.rm("abfss://<container>@<account>.dfs.core.windows.net/<path>", recurse=True)

Then recreate the table

CREATE OR REPLACE TABLE catalog.schema.table
USING DELTA
LOCATION 'abfss://<container>@<account>.dfs.core.windows.net/<path>'
AS SELECT * FROM some_source;

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

szymon_dybczak — Wed, 17 Sep 2025 14:15:37 GMT

Hi @Carlton ,

This could happen for following reasons:

Manually remove underlying files from the Delta log.
Run rm commands or other non-Delta operations that remove files from the Delta log.
Drop and immediately create a table on top of the same location.

When files are manually removed or not removed correctly, the Delta log versions become non-contiguous.

You can try to follow this guide to resolve this issue:

Dropping and recreating Delta tables results in a DeltaVersionsNotContiguousException error - Databricks

Also, the way you were doing in (like manually deleting files) is not recommended by Databricks.
Instead of dropping and recreating Delta tables, the official recommendation is to use CREATE OR REPLACE command.

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

K_Anudeep — Wed, 17 Sep 2025 14:27:26 GMT

Hello @Carlton ,

Good Day!

What is the DBR version? How are you creating the table? Did you drop the table and recreate it using SQL?

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 14:28:52 GMT

Hi Guys,

I really appreciate you getting in touch, this has been driving me crazy.

I delete the entire folder from ADLS. Admittedly, I didn't remove the folder from within Databricks using dbutils.fs.rm.

If you take a look at my image, I want to remove the 'Country'. So I would delete the Country folder and all sub-folders e.g. folder 1 and then _delta_log folder. But when attempt to create another Country deltaTable I get the error.

Are you suggesting that I should delete from within databricks?

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 14:30:00 GMT

Sorry I forgot to add the image:

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

K_Anudeep — Wed, 17 Sep 2025 14:33:15 GMT

Hi @Carlton ,

Yes—delete from inside Databricks (and drop the table) when you plan to reuse the same path/name. External deletes can leave you with a half-cleaned path or stale state, which leads to the “protocol could not be recovered (version: 0)” error.

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 14:56:13 GMT

Hi nayan_wylde, I'm getting the following error when I try to list the files in account

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

szymon_dybczak — Wed, 17 Sep 2025 15:02:10 GMT

Hi,

Delete all folders in ADLS related to a country and also execute DROP TABLE command on the catalog where this table resides.

Probably catalog still stores metadata about this table and thinks it exists.

Then try to create table - but only if you have performed above steps

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 15:08:56 GMT

Hi szymon_dybczak, if you take a look at the image, are you suggesting I delete the folder BASE? That folder is the equivalent to our SILVER layer. Deleting that folder would be devastating to our business. I deleted from DATA and all sub-folders below DATA including Country. Shouldn't that be enough?

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

szymon_dybczak — Wed, 17 Sep 2025 15:26:58 GMT

Hi, sorry. I should be more specific. You should delete country folder along with all subfolders ( basically the folder in which the data for the table you want to delete resides).

After that, execute DROP TABLE command on the catalog where the table you wanted to delete was registered.

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 15:34:12 GMT

Hi szymon_dybczak, no worries, I'm just grateful that you're helping me out. I did what you suggested along with deleting the table from the catalog, see image but I'm still getting the same error

I should mention that I'm getting the following error when I try list files

display(dbutils.fs.ls("abfss://root@xxxxxx.dfs.core.windows.net/"))

fs.azure.account.keyInvalid configuration value detected for fs.azure.account.key

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 15:47:37 GMT

I should also mention that not only do I get the error:

DeltaIllegalStateException: The protocol of your Delta table could not be recovered while Reconstructing version: 0. Did you manually delete files in the _delta_log directory?

The table folder that is created looks like the following:

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 15:53:40 GMT

I should also mentioned I am using the following code to create the deltaTable which I have been using for years

deltadf = DeltaTable.forName(spark,f"{stageName}{regName}")

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

szymon_dybczak — Wed, 17 Sep 2025 16:00:24 GMT

Thanks for additional details. Could you also tell us if you're using Hive metastore or Unity Catalog?

If you are using Hive then how you have configured storage?

Also, do you see something useful in driver logs?

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 16:10:07 GMT

Hi @szymon_dybczak at the moment, I'm using Databricks Community Edition, however I get the same issue when I'm using a premium edition of Databricks

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

nayan_wylde — Wed, 17 Sep 2025 16:33:12 GMT

@Carlton This error means you are not passing the client id and client secret in the spark config that will authenticate against the storage or the client secret expired.

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 16:45:45 GMT

Any more thoughts on this issue guys

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

szymon_dybczak — Wed, 17 Sep 2025 17:50:30 GMT

So, since you're using databricks community edition I'm assuming that you're dealing with Hive metastore.
You can try following:

1. Go to Settings

2. Now go to Advance (2) and turn on DBFS File Browser (3)

3. Now, click Catalog (4), then DBFS(5) and go to users -> hive -> warehouse and check if you can find the table that you want to delete (Country).

4. If the table is there, copy path to the table (just like I did for flattened_source table in screen below)

5. Then remove all files using dbutils.fm.rm

dbutils.fs.rm(dir="dbfs:/user/hive/warehouse/invoices_bz", recurse=True)

6. Double check that there are no files in your ADLS related to Country table

7. Try to recreate it.

Re: No Longer Able to Create DeltaTables in ADLS Gen 2

Carlton — Wed, 17 Sep 2025 18:03:54 GMT

Hi @szymon_dybczak , you must have a special version of Databricks Community Edition as I don't have those options after I select settings

Shall I try from my premium paid for version of Databricks?