cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How can I run a streaming query on a new table with tbl property: change data feed enabled?

nimble
New Contributor

In Databricks on AWS, I am trying to run a streaming query (trigger=Once) with delta.enableChangeDataFeed=true in the table definition as instructed, but this always fails with :

ERROR: Some streams terminated before this command could finish!
 
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Error getting change data for range [0 , 0] as change data was not recorded for version [0]. If you\'ve enabled change data feed on this table, use `DESCRIBE HISTORY` to see when it was first enabled.
Otherwise, to start recording change data, use `ALTER TABLE table_name SET TBLPROPERTIES
(delta.enableChangeDataFeed=true)`.

I tried:

  • removing checkpoints for streaming query
  • recreating the table
  • adding "spark.databricks.delta.properties.defaults.enableChangeDataFeed": "true" to spark_conf of the compute

None of those fixed the issue. This is quite puzzling as this setting works for other tables.

2 REPLIES 2

Hubert-Dudek
Esteemed Contributor III

What is precisely recreating a table? Maybe delta files are in the old version, which is not supporting CDC. Please also share your code.

Please first just read CDC not as a stream but as SELECT * FROM table_changes(table)

swethaNandan
New Contributor III
New Contributor III

Hi @daniel e​ 

Can you try running the select command on table changes from 0th version and see if you get output?

SELECT * FROM table_changes('tableName', 0)

Also, Please share the streaming query that you are running.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!