topic How to maintain Primary Key Column in Databricks Delta Multi Cluster environment in Data Engineering

How to maintain Primary Key Column in Databricks Delta Multi Cluster environment

AdityaDeshpande — Sun, 25 Aug 2019 11:47:41 GMT

I am trying to replicate the SQL DB like feature of maintaining the Primary Keys in Databrciks Delta approach where the data is being written to Blob Storage such as ADLS2 oe AWS S3.

I want a Auto Incremented Primary key feature using Databricks Delta.

Existing approach - is using the latest row count and maintaining the Primary keys. However, this approach does not suit in parallel processing environment where Primary keys get duplicated data.

Re: How to maintain Primary Key Column in Databricks Delta Multi Cluster environment

girivaratharaja — Mon, 26 Aug 2019 14:13:23 GMT

Hi @Aditya Deshpande

There is no locking mechanism of PK in Delta. You can use row_number() function on the df and save using delta and do a distinct() before the write.

Re: How to maintain Primary Key Column in Databricks Delta Multi Cluster environment

AdityaDeshpande — Tue, 27 Aug 2019 07:09:49 GMT

This is the existing approach we're using. But this has limitation when run in Multi Cluster environment which is writing the data into same Destination.