3 weeks ago
Hello,
I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.
So, as per the Microsoft migration pre-requisites:
File or directory names with only spaces or tabs, ending with a ., containing a :, or with multiple consecutive forward slashes (//) aren't compatible with Gen2. You need to rename these files or directories before you migrate.
Delta tables were created on the partition fields that contained dot (.), so the partition folder path name has dot(.) as well and hence unable to migrate it to ADLS Gen2.
Example :
/Fruits/Mango/Alfonso/DATE=20200101/LOCATION=UK/STORE=S.K DALTON/part-32434-df7cge3e-4201-4c47-83f1-ef034c34543b.c000.snappy.parquet
Need help on what could be workaround for this so without loosing data, I can migrate to ADLS Gen2. Also ensure history is maintained when doing version check.
Thanks
3 weeks ago
Hi @hold_my_samosa, Certainly! Migrating Delta tables from Azure ADLS Gen1 to Gen2 can be tricky, especially when dealing with partition folder path names that contain dots.
Letโs explore some workarounds and ensure data integrity during the migration:
Understanding the Issue:
Workaround Options:
Data History and Versioning:
Additional Considerations:
Remember to test these approaches in a controlled environment before applying them to your production data. Good luck with your migration!
3 weeks ago
Please recommend other solutions if possible as above provided may not work due to below reasons.
Data History and Versioning:
Additional Considerations:
2 weeks ago - last edited 2 weeks ago
Hi @hold_my_samosa , Could you please explain what exactly is the issue now? What works and what doesn't?
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.