cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks-Connect and Change Data Feed query error

Ian
New Contributor III

I have installed Databricks-Connect (9.1 LTS). I am able to send queries to the cluster. However, when the query includes a call to the 'table_changes' function that is a part of Change Data Feed, I get the following error:

AnalysisException("could not resolve `table_changes` to a table-valued function; line 1 

The query runs when I run it in a notebook against the same cluster. The query looks similar to the following:

select distinct SomeFieldID from table_changes('SomeTable','2021-12-31 9:53:02') where _change_type in ('update_postimage')

1 ACCEPTED SOLUTION

Accepted Solutions

Hi @Ian Corera​ ,

Please check the limitations from DBconnect. https://docs.databricks.com/dev-tools/databricks-connect.html#limitations It seems like the feature work fine on the notebook but not on DBconnect.

View solution in original post

6 REPLIES 6

Anonymous
Not applicable

Hello @Ian Corera​ - My name is Piper, and I'm one of the moderators for Databricks. Let's see what your fellow members have to say before we circle back around to this. 🙂

Kaniz_Fatma
Community Manager
Community Manager

Hi @Ian Corera, Can you paste the snap of your "table_changes" function code?​ 

Ian
New Contributor III

Hi @Kaniz Fatma​ , the table_changes function is an internal Databricks function used in Change Data Feed (CDF).

Please refer to the article below. It discusses the table_changes function.

https://docs.databricks.com/delta/delta-change-data-feed.html

Kaniz_Fatma
Community Manager
Community Manager

Hi @Ian Corera​ , Delta change data feed is not enabled by default. You must explicitly enable the change data feed option using one of the following methods:

1. New table: Set the table property delta.enableChangeDataFeed = true in the CREATE TABLE command.

CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIES (delta.enableChangeDataFeed = true)

2. Existing table: Set the table property delta.enableChangeDataFeed = true in the ALTER TABLE command.

ALTER TABLE myDeltaTable SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

3. All new tables:

set spark.databricks.delta.properties.defaults.enableChangeDataFeed = true;

Source

Ian
New Contributor III

Hi @Kaniz Fatma​  thank you for your reply. I do have the tables setup for Change Data Feed. I am able to run the query with 'table_changes' in a notebook. However, the same query does not run in Databricks-Connect (9.1 LTS).

In Databricks-Connect I get the error below:

AnalysisException("could not resolve `table_changes` to a table-valued function; line 1 

Again, the query runs fine in a notebook in a Databricks workspace.

Hi @Ian Corera​ ,

Please check the limitations from DBconnect. https://docs.databricks.com/dev-tools/databricks-connect.html#limitations It seems like the feature work fine on the notebook but not on DBconnect.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!