topic Re: Databricks S3 Commit Service in Data Engineering

Databricks S3 Commit Service

ed_carv — Mon, 02 Sep 2024 10:35:06 GMT

Is Databricks S3 Commit Service enabled by default if Unity Catalog is not enabled and the compute resources run in our AWS account (classic compute plane)? If not, how can it be enabled?

This service seems to resolve the limitations with multi-cluster write to Delta Lake tables stored in S3 to guarantee ACID transactions.

I understand this Delta Lake limitation can also be resolved by setting up DynamoDB for delta logs, but wanted to confirm if this is still necessary as it seems Databricks has its own solution for this problem.

Re: Databricks S3 Commit Service

VZLA — Fri, 27 Dec 2024 17:38:23 GMT

No, the Databricks S3 commit service is not guaranteed to be enabled by default in the AWS classic compute plane. The configuration may vary based on your specific workspace setup.

How can it be enabled?

To enable the Databricks S3 commit service, follow these steps:

Ensure proper instance profiles are configured to grant clusters appropriate access to S3 buckets.
Configure Spark parameters to explicitly enable the service and disable conflicting optimizations like direct uploads.

https://docs.databricks.com/en/security/network/classic/s3-commit-service.html