Parquet Encryption/Decryption in Databricks

[Aya-Ahmed]

Hi everyone,

I'm curious about Databricks' approach to encrypting and decrypting Parquet files. Does Databricks adhere to standard encryption/decryption methods for Parquet? If not, what specific methods or techniques are used?

I'd appreciate any insights or recommendations on best practices for securing Parquet data in Databricks.

Thanks!

[Witold]

Since Databricks uses Spark, you should be able to use e.g. Columnar Encryption

Besides, you can look into this and aes-specific functions.

[Aya-Ahmed]

Thank you for your reply!

As I understand, they don’t support columnar encryption automatically; we need to add some logic manually. Is this related to a performance issue that columnar encryption level would cause? What I mean, do they recommend columnar encryption?