Databricks S3 Commit Service

[ed_carv]

Is Databricks S3 Commit Service enabled by default if Unity Catalog is not enabled and the compute resources run in our AWS account (classic compute plane)? If not, how can it be enabled?

This service seems to resolve the limitations with multi-cluster write to Delta Lake tables stored in S3 to guarantee ACID transactions.

I understand this Delta Lake limitation can also be resolved by setting up DynamoDB for delta logs, but wanted to confirm if this is still necessary as it seems Databricks has its own solution for this problem.

[VZLA]

No, the Databricks S3 commit service is not guaranteed to be enabled by default in the AWS classic compute plane. The configuration may vary based on your specific workspace setup.

How can it be enabled?

To enable the Databricks S3 commit service, follow these steps:

1. Ensure proper instance profiles are configured to grant clusters appropriate access to S3 buckets.
2. Configure Spark parameters to explicitly enable the service and disable conflicting optimizations like direct uploads.

https://docs.databricks.com/en/security/network/classic/s3-commit-service.html