Hi,
When reading Delta Lake file (created by Auto Loader) with this code: df = (
spark.readStream
.format('cloudFiles')
.option("cloudFiles.format", "delta")
.option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint")
.load(bronze_path)
)
Receives this error:
AnalysisException: Incompatible format detected. A transaction log for Delta was found at `/mnt/f1/f2/_delta_log`, but you are trying to read from `/mnt/f1/f2/` using format("cloudFiles"). You must use 'format("delta")' when reading and writing to a delta table. To disable this check, SET spark.databricks.delta.formatCheck.enabled=false To learn more about Delta...
What's right way of reading Delta Lake files with Auto Loader for further processing (e.g.. from Bronze layer to Silver)?
Thank you!