Databricks-Connect and Change Data Feed query error

[Ian]

I have installed Databricks-Connect (9.1 LTS). I am able to send queries to the cluster. However, when the query includes a call to the 'table_changes' function that is a part of Change Data Feed, I get the following error:

AnalysisException("could not resolve `table_changes` to a table-valued function; line 1 

The query runs when I run it in a notebook against the same cluster. The query looks similar to the following:

select distinct SomeFieldID from table_changes('SomeTable','2021-12-31 9:53:02') where _change_type in ('update_postimage')

[Anonymous]

Hello @Ian Corera​ - My name is Piper, and I'm one of the moderators for Databricks. Let's see what your fellow members have to say before we circle back around to this. 🙂

[Kaniz_Fatma]

Hi @Ian Corera, Can you paste the snap of your "table_changes" function code?​ 

[Ian]

Hi @Kaniz Fatma​ , the table_changes function is an internal Databricks function used in Change Data Feed (CDF).

Please refer to the article below. It discusses the table_changes function.

https://docs.databricks.com/delta/delta-change-data-feed.html

[Kaniz_Fatma]

Hi @Ian Corera​ , Delta change data feed is not enabled by default. You must explicitly enable the change data feed option using one of the following methods:

1. New table: Set the table property delta.enableChangeDataFeed = true in the CREATE TABLE command.

CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIES (delta.enableChangeDataFeed = true)

2. Existing table: Set the table property delta.enableChangeDataFeed = true in the ALTER TABLE command.

ALTER TABLE myDeltaTable SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

3. All new tables:

set spark.databricks.delta.properties.defaults.enableChangeDataFeed = true;

Source

[Ian]

Hi @Kaniz Fatma​  thank you for your reply. I do have the tables setup for Change Data Feed. I am able to run the query with 'table_changes' in a notebook. However, the same query does not run in Databricks-Connect (9.1 LTS).

In Databricks-Connect I get the error below:

AnalysisException("could not resolve `table_changes` to a table-valued function; line 1 

Again, the query runs fine in a notebook in a Databricks workspace.

[jose_gonzalez · Accepted solution]

Hi @Ian Corera​ ,

Please check the limitations from DBconnect. https://docs.databricks.com/dev-tools/databricks-connect.html#limitations It seems like the feature work fine on the notebook but not on DBconnect.