How can I run a streaming query on a new table with tbl property: change data feed enabled?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-06-2022 03:35 AM
In Databricks on AWS, I am trying to run a streaming query (trigger=Once) with delta.enableChangeDataFeed=true in the table definition as instructed, but this always fails with :
ERROR: Some streams terminated before this command could finish!
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Error getting change data for range [0 , 0] as change data was not recorded for version [0]. If you\'ve enabled change data feed on this table, use `DESCRIBE HISTORY` to see when it was first enabled.
Otherwise, to start recording change data, use `ALTER TABLE table_name SET TBLPROPERTIES
(delta.enableChangeDataFeed=true)`.I tried:
- removing checkpoints for streaming query
- recreating the table
- adding "spark.databricks.delta.properties.defaults.enableChangeDataFeed": "true" to spark_conf of the compute
None of those fixed the issue. This is quite puzzling as this setting works for other tables.
Labels: