Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2025 09:59 PM
Databricks has a special DBIO protocol that uses the _started and _committed files to transactionally write to cloud storage.
You can disable this by setting the below spark config
spark.conf.set("spark.sql.sources.commitProtocolClass", "org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol")
Also, you can read more about DBIO here