How can I configure S3 Client-Side Encryption (CSE-KMS ) for my data pipeline

[sajith_appukutt · Accepted solution]

You could create a  custom key provider that implements the EncryptionMaterialsProvider interface and configure the databricks mount to use CSE

dbutils.fs.mount(
  "s3a://cse-bucket",
  "/mnt/cse-data", 
  extraConfigs = Map(
    "fs.s3.cse.enabled" -> "true",
    "fs.s3.cse.encryptionMaterialsProvider" -> "com.mynamespace. MyEncryptionMaterialsProviders",
    "kms.key.id" -> "xxx"
  )
)

[AdrianRojas]

a bit old, but I just faced the same issue, specifying a custom EncryptionMaterialsProvider (as described in the previous post) did the trick for me but I did had to also specify my kms endpoint, just because my region:

"fs.s3.cse.kms.endpoint" -> "kms.<region>.amazonaws.com"