How to maintain Primary Key Column in Databricks Delta Multi Cluster environment
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-25-2019 04:47 AM
I am trying to replicate the SQL DB like feature of maintaining the Primary Keys in Databrciks Delta approach where the data is being written to Blob Storage such as ADLS2 oe AWS S3.
I want a Auto Incremented Primary key feature using Databricks Delta.
Existing approach - is using the latest row count and maintaining the Primary keys. However, this approach does not suit in parallel processing environment where Primary keys get duplicated data.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-26-2019 07:13 AM
Hi @Aditya Deshpande
There is no locking mechanism of PK in Delta. You can use row_number() function on the df and save using delta and do a distinct() before the write.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-27-2019 12:09 AM
This is the existing approach we're using. But this has limitation when run in Multi Cluster environment which is writing the data into same Destination.
