How can I run a streaming query on a new table with tbl property: change data feed enabled?

nimble
New Contributor

In Databricks on AWS, I am trying to run a streaming query (trigger=Once) with delta.enableChangeDataFeed=true in the table definition as instructed, but this always fails with :

ERROR: Some streams terminated before this command could finish!
 
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Error getting change data for range [0 , 0] as change data was not recorded for version [0]. If you\'ve enabled change data feed on this table, use `DESCRIBE HISTORY` to see when it was first enabled.
Otherwise, to start recording change data, use `ALTER TABLE table_name SET TBLPROPERTIES
(delta.enableChangeDataFeed=true)`.

I tried:

  • removing checkpoints for streaming query
  • recreating the table
  • adding "spark.databricks.delta.properties.defaults.enableChangeDataFeed": "true" to spark_conf of the compute

None of those fixed the issue. This is quite puzzling as this setting works for other tables.