How to maintain Primary Key Column in Databricks Delta Multi Cluster environment

[AdityaDeshpande]

I am trying to replicate the SQL DB like feature of maintaining the Primary Keys in Databrciks Delta approach where the data is being written to Blob Storage such as ADLS2 oe AWS S3.

I want a Auto Incremented Primary key feature using Databricks Delta.

Existing approach - is using the latest row count and maintaining the Primary keys. However, this approach does not suit in parallel processing environment where Primary keys get duplicated data.

[girivaratharaja]

Hi @Aditya Deshpande​ 

There is no locking mechanism of PK in Delta. You can use row_number() function on the df and save using delta and do a distinct() before the write.

[AdityaDeshpande]

This is the existing approach we're using. But this has limitation when run in Multi Cluster environment which is writing the data into same Destination.