Delta Partition File on Azure ADLS Gen2 Migration

[hold_my_samosa]

Hello,

I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.

So, as per the Microsoft migration pre-requisites:

File or directory names with only spaces or tabs, ending with a ., containing a :, or with multiple consecutive forward slashes (//) aren't compatible with Gen2. You need to rename these files or directories before you migrate.

Delta tables were created on the partition fields that contained dot (.), so the partition folder path name has dot(.) as well and hence unable to migrate it to ADLS Gen2.

Example : 

/Fruits/Mango/Alfonso/DATE=20200101/LOCATION=UK/STORE=S.K DALTON/part-32434-df7cge3e-4201-4c47-83f1-ef034c34543b.c000.snappy.parquet

Need help on what could be workaround for this so without loosing data, I can migrate to ADLS Gen2. Also ensure history is maintained when doing version check.

Thanks

[hold_my_samosa]

Please recommend other solutions if possible as above provided may not work due to below reasons.

Data History and Versioning:

• To maintain history during the migration, ensure that your Delta table retains its version history. --> To maintain history, it has to keep those folder name with dot.
• Delta Lake automatically tracks changes, so you’ll have access to historical data even after the migration. --> only if migration is possible which is not as folder name has dot.

Additional Considerations:

• Verify that your workspace is enabled for Unity Catalog (if needed). --> Yes it is.
• Confirm that your clusters are running Databricks Runtime 11.3 LTS or above to fully support Unity Catalog features. --> yes they are.