Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2023 03:45 AM
Thank you for a comprehensive reply. I have comments to each point below. For now, we have actually decided to put off the move so it is not an imminent issue, though it will come back.
The basic issue is related to delta tables, and the fact that timestamps on the files themselves (as opposed to file contents that include timestamps)
- Azure Databricks Mount Points
- This only affects how data is accessed through Databricks, and doesn't deal with the actual moving of data between accounts.
- Azure Service Principal and OAuth 2.0
- This is indeed how we connect to the storage. But again, this doesn't address the issue of moving data between accounts.
- Automated Data Migration
- This addresses the data movement issue, and could be interesting to learn more about. It all depends on whether "allow you to maintain timestamps and minimize manual intervention" is true for the specific timestamps that delta consumer depends on. If you have more specifics on this it would be helpful to anyone with similar issue. I found the following statement which seems to suggest this is not simple: https://learn.microsoft.com/en-us/answers/questions/656437/is-it-possible-to-edit-the-lastmodified-a...
- Delta Lake Table Copy
- COPY INTO "Loads data from a file location into a Delta table.", and accepts the formats CSV, JSON, AVRO, ORC, PARQUET, TEXT, BINARYFILE. This does not seem to address the basic issue
- We were also considering deep clone, but according to Databricks documentation "A cloned table has an independent history from its source table. Time travel queries on a cloned table do not work with the same inputs as they work on its source table."
