Delta Partition File on Azure ADLS Gen2 Migration

[hold_my_samosa]

Hello,

I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.

So, as per the Microsoft migration pre-requisites:

File or directory names with only spaces or tabs, ending with a ., containing a :, or with multiple consecutive forward slashes (//) aren't compatible with Gen2. You need to rename these files or directories before you migrate.

Delta tables were created on the partition fields that contained dot (.), so the partition folder path name has dot(.) as well and hence unable to migrate it to ADLS Gen2.

Example : 

/Fruits/Mango/Alfonso/DATE=20200101/LOCATION=UK/STORE=S.K DALTON/part-32434-df7cge3e-4201-4c47-83f1-ef034c34543b.c000.snappy.parquet

Need help on what could be workaround for this so without loosing data, I can migrate to ADLS Gen2. Also ensure history is maintained when doing version check.

Thanks

[Kaniz]

Hi @hold_my_samosa, Certainly! Migrating Delta tables from Azure ADLS Gen1 to Gen2 can be tricky, especially when dealing with partition folder path names that contain dots. 

Let’s explore some workarounds and ensure data integrity during the migration:

Understanding the Issue:

• The error message you encountered is due to the restrictions imposed by Azure ADLS Gen2 on file and directory names.
• Specifically, folder names with dots (.) are not compatible with Gen2.

Workaround Options:

• Here are some approaches you can consider:

Data History and Versioning:

• To maintain history during the migration, ensure that your Delta table retains its version history.
• Delta Lake automatically tracks changes, so you’ll have access to historical data even after the migration.

Additional Considerations:

• Verify that your workspace is enabled for Unity Catalog (if needed).
• Confirm that your clusters are running Databricks Runtime 11.3 LTS or above to fully support Unity Catalog features.

Remember to test these approaches in a controlled environment before applying them to your production data. Good luck with your migration!

[hold_my_samosa]

Please recommend other solutions if possible as above provided may not work due to below reasons.

Data History and Versioning:

• To maintain history during the migration, ensure that your Delta table retains its version history. --> To maintain history, it has to keep those folder name with dot.
• Delta Lake automatically tracks changes, so you’ll have access to historical data even after the migration. --> only if migration is possible which is not as folder name has dot.

Additional Considerations:

• Verify that your workspace is enabled for Unity Catalog (if needed). --> Yes it is.
• Confirm that your clusters are running Databricks Runtime 11.3 LTS or above to fully support Unity Catalog features. --> yes they are.

[Kaniz]

Hi @hold_my_samosa , Could you please explain what exactly is the issue now? What works and what doesn't?