Azure Databricks, migrating delta table data with CDF on.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-21-2023 02:44 AM
We are on Azure Databricks over ADLS Gen2 and have a set of tables and workflows that process data from and between those tables, using change data feeds. (We are not yet using Unity Catalog, and also not Hive metastore, just accessing delta tables from storage).
Now we are required by IT to move data to a different Azure storage account, and I am faced with a conundrum. Delta documentation states that it is important to maintain timestamps if you want to preserve CDF. https://docs.delta.io/latest/delta-faq.html#can-i-copy-my-delta-lake-table-to-another-location
On the other hand, Azure support rep states that the only way they can think of is to manually use Storage Explorer something like this:
On Storage Explorer
Export the table as CSV, with option to export the timestamp
Import the CSV into the the new Storage Account changing the name of the Timestamp to something else, like Timestamp2.
Which will add a new field in the new table and also consumes more space because of it.
I need to try that recipe to see if it actually makes sense, but in reality we have many tables and such a manual process would be inefficient and error-prone.
Maybe someone here knows a better way?
- Labels:
-
Delta Lake
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-21-2023 04:45 AM
As it turns out, due to a misunderstanding, the responses from Azure support were answering a slightly different question (about Azure Table Storage instead of Delta Tables on Blob/ADLS Gen2), so we'll try there again. However, still interested in ideas/experiences from this community.
Also, I couldn't find a way to fix the original post, or delete and recreate without the confusing details that turned out to be about Azure Table Storage so sorry about that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2023 03:45 AM