cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Partition File on Azure ADLS Gen2 Migration

hold_my_samosa
New Contributor II

Hello,

I am working on a migration project and I am facing issue while migrating delta tables from Azure ADLS Gen1 to Gen2.

So, as per the Microsoft migration pre-requisites:

File or directory names with only spaces or tabs, ending with a ., containing a :, or with multiple consecutive forward slashes (//) aren't compatible with Gen2. You need to rename these files or directories before you migrate.

Delta tables were created on the partition fields that contained dot (.), so the partition folder path name has dot(.) as well and hence unable to migrate it to ADLS Gen2.

Example : 

/Fruits/Mango/Alfonso/DATE=20200101/LOCATION=UK/STORE=S.K DALTON/part-32434-df7cge3e-4201-4c47-83f1-ef034c34543b.c000.snappy.parquet

Need help on what could be workaround for this so without loosing data, I can migrate to ADLS Gen2. Also ensure history is maintained when doing version check.

Thanks

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @hold_my_samosa, Certainly! Migrating Delta tables from Azure ADLS Gen1 to Gen2 can be tricky, especially when dealing with partition folder path names that contain dots. 

 

Let’s explore some workarounds and ensure data integrity during the migration:

 

Understanding the Issue:

  • The error message you encountered is due to the restrictions imposed by Azure ADLS Gen2 on file and directory names.
  • Specifically, folder names with dots (.) are not compatible with Gen2.

Workaround Options:

  • Here are some approaches you can consider:

Data History and Versioning:

  • To maintain history during the migration, ensure that your Delta table retains its version history.
  • Delta Lake automatically tracks changes, so you’ll have access to historical data even after the migration.

Additional Considerations:

  • Verify that your workspace is enabled for Unity Catalog (if needed).
  • Confirm that your clusters are running Databricks Runtime 11.3 LTS or above to fully support Unity Catalog features.

Remember to test these approaches in a controlled environment before applying them to your production data. Good luck with your migration!

Please recommend other solutions if possible as above provided may not work due to below reasons.

Data History and Versioning:

  • To maintain history during the migration, ensure that your Delta table retains its version history. --> To maintain history, it has to keep those folder name with dot.
  • Delta Lake automatically tracks changes, so you’ll have access to historical data even after the migration. --> only if migration is possible which is not as folder name has dot.

Additional Considerations:

  • Verify that your workspace is enabled for Unity Catalog (if needed). --> Yes it is.
  • Confirm that your clusters are running Databricks Runtime 11.3 LTS or above to fully support Unity Catalog features. --> yes they are.

Kaniz_Fatma
Community Manager
Community Manager

Hi @hold_my_samosa , Could you please explain what exactly is the issue now? What works and what doesn't?

 

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group