Databricks-Autoloader-S3-KMS
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-14-2023 12:46 AM
Hi, I am working on a requirement where I am using autoloader in a DLT pipeline to ingest new files as they come.
This flow is working fine. However I am facing an issue, when we have the source bucket an s3 location, since the bucket is having a SSE-KMS setup and when it tries to write schemaLocation, it is throwing an error.
What extra changes need to be done, I have provided the kms related permissions to the instance profile that is getting used. What changes I need to make in my notebook/pipeline to get it working.