topic encryption in Data Engineering

encryption

Phani1 — Mon, 20 Nov 2023 16:57:13 GMT

Hi Databricks, Could you please guide me on the below scenario?

Here is the use case we are trying to solve for

Currently environment is using “Voltage” as an encryption tool for encrypting the data in S3 in conjunction with business business-provided data encryption & masking catalog.
We are looking to replace the above tool, with inherent Databricks capabilities if possible and yet apply 256-bit AES encryption standards.

Scenario-

Is it possible to apply data encryption at S3 bucket where Schema & Catalog is maintained by using Customer_managed Key - need to solve for this.
Once encryption is done, we need to then read the data into Databricks DF using custom UDFs, where standard functions like- “aes_encrypt/ aes_decrypt” function could be used along with the data masking functionalities of Unity Catalog.

Re: encryption

DeborahRoe — Wed, 22 Nov 2023 14:13:55 GMT

Thanks for the clear guidance! Integrating AWS KMS for CMK in Databricks and using custom UDFs for encryption/decryption align perfectly with our goals. Appreciate your assistance!

Re: encryption

AliaCollier — Wed, 06 Dec 2023 08:12:38 GMT

To replace "Voltage" with Databricks encryption, follow these steps: set up a Customer Managed Key in AWS, configure the S3 bucket, read data in Databricks, and implement custom UDFs for AES encryption/decryption.