topic Re: Delta table partition directories when column mapping is enabled in Data Engineering

Delta table partition directories when column mapping is enabled

Gary_Irick — Tue, 13 Sep 2022 18:20:21 GMT

I recently created a table on a cluster in Azure running Databricks Runtime 11.1. The table is partitioned by a "date" column. I enabled column mapping, like this:

ALTER TABLE {schema}.{table_name} SET TBLPROPERTIES('delta.columnMapping.mode' = 'name', 'delta.minReaderVersion' = '2', 'delta.minWriterVersion' = '5')

Before enabling column mapping, the directory containing the Delta table has the expected partition directories: "date=2022-08-18", "date=2022-08-19", etc.

After enabling column mapping, every time I do a MERGE into that table, I get new directories created with short names like "5k", "Rw", "Yd", etc. When I VACUUM the table, most of the directories are empty, but the empty directories are not removed. We merge into this table frequently, so the table containing the Delta table ends up with lots and lots of empty directories.

I have 2 questions:

Is it expected that these directories will be created with names other than the expected "date=2022-08-18"?

Is there a way to make VACUUM remove the empty directories?

I could write code to walk through the Delta table directory and remove the empty directories, but I would rather not touch those directories! That's for Databricks to manage, and I don't want to step in its way.

Thanks in advance for any information you can provide.

Re: Delta table partition directories when column mapping is enabled

Anonymous — Tue, 27 Sep 2022 12:11:31 GMT

Hi @Gary Irick

Does @Debayan Mukherjee response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!

Re: Delta table partition directories when column mapping is enabled

gongasxavi — Fri, 16 Dec 2022 17:26:06 GMT

The same is happening with me. Since enabling column mapping, the new records are stored in folders with random names instead of being stored in its partition folder

Re: Delta table partition directories when column mapping is enabled

Pete_Cotton — Tue, 03 Jan 2023 12:00:18 GMT

Same issue is happening with me too since enabling column mapping. Files are stored in folders with random 2 character names (0P, 3h, BB) rather than the date value of the load_date partition column (load_date=2023-01-01, load_date=2023-01-02).

Have tried using databricks runtime 12.0 but get the same result when performing an append or merge operation. Has anyone been able to resolve this yet?

Re: Delta table partition directories when column mapping is enabled

AleksAngelova — Tue, 04 Apr 2023 11:51:14 GMT

Is there at least an explanation why this is happening and whether it affects performance?

Re: Delta table partition directories when column mapping is enabled

Debayan — Fri, 16 Sep 2022 04:55:52 GMT

Hi, For removing files or directories using VACUUM , you can refer https://docs.databricks.com/delta/delta-utility.html#remove-files-no-longer-referenced-by-a-delta-table

As far as I know, the dates will be the default naming syntax, which can be renamed.

Re: Delta table partition directories when column mapping is enabled

nan — Wed, 12 Jul 2023 19:37:17 GMT

seen the same behavior. waiting for some explanation.

Re: Delta table partition directories when column mapping is enabled

Tharun-Kumar — Wed, 12 Jul 2023 20:52:21 GMT

@Gary_Irick @Pete_Cotton
This is expected. Enabling column mapping enables random file prefixes, which removes the ability to explore data using Hive-style partitioning.

This is also documented here - https://docs.databricks.com/delta/delta-column-mapping.html#:~:text=Enabling%20column%20mapping%20also%20enables%20random%20file%20prefixes%2C%20which%20removes%20the%20ability%20to%20explore%20data%20using%20Hive%2Dstyle%20partitioning.

Re: Delta table partition directories when column mapping is enabled

vascoa — Mon, 20 Nov 2023 16:48:11 GMT

Same is happening to me and very frustrating as it irreversibly breaks our process.

Re: Delta table partition directories when column mapping is enabled

talenik — Tue, 30 Jul 2024 10:13:52 GMT

Hi @Retired_mod ,

I have few queries on Directory Names with Column Mapping. I have this delta table on ADLS and I am trying to read it, but I am getting below error. How can we read delta tables with column mapping enabled with pyspark?

Can you please help.

A partition path fragment should be the form like `part1=foo/part2=bar`. The partition path: {{delta table name}}

Edit:

I was able to read tables as is. Maybe some issue with delta version

Regards,

Nikhil

Re: Delta table partition directories when column mapping is enabled

Narsikakunuri — Mon, 19 Jan 2026 16:59:05 GMT

Still same behaviour when Column Mapping enabled