01-03-2022 10:49 AM
I have installed Databricks-Connect (9.1 LTS). I am able to send queries to the cluster. However, when the query includes a call to the 'table_changes' function that is a part of Change Data Feed, I get the following error:
AnalysisException("could not resolve `table_changes` to a table-valued function; line 1
The query runs when I run it in a notebook against the same cluster. The query looks similar to the following:
select distinct SomeFieldID from table_changes('SomeTable','2021-12-31 9:53:02') where _change_type in ('update_postimage')
02-08-2022 04:58 PM
Hi @Ian Corera ,
Please check the limitations from DBconnect. https://docs.databricks.com/dev-tools/databricks-connect.html#limitations It seems like the feature work fine on the notebook but not on DBconnect.
01-03-2022 03:17 PM
Hello @Ian Corera - My name is Piper, and I'm one of the moderators for Databricks. Let's see what your fellow members have to say before we circle back around to this. 🙂
01-20-2022 10:06 PM
Hi @Ian Corera, Can you paste the snap of your "table_changes" function code?
01-21-2022 11:00 AM
Hi @Kaniz Fatma , the table_changes function is an internal Databricks function used in Change Data Feed (CDF).
Please refer to the article below. It discusses the table_changes function.
https://docs.databricks.com/delta/delta-change-data-feed.html
01-24-2022 04:53 AM
Hi @Ian Corera , Delta change data feed is not enabled by default. You must explicitly enable the change data feed option using one of the following methods:
1. New table: Set the table property delta.enableChangeDataFeed = true in the CREATE TABLE command.
CREATE TABLE student (id INT, name STRING, age INT) TBLPROPERTIES (delta.enableChangeDataFeed = true)
2. Existing table: Set the table property delta.enableChangeDataFeed = true in the ALTER TABLE command.
ALTER TABLE myDeltaTable SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
3. All new tables:
set spark.databricks.delta.properties.defaults.enableChangeDataFeed = true;
01-24-2022 05:44 AM
Hi @Kaniz Fatma thank you for your reply. I do have the tables setup for Change Data Feed. I am able to run the query with 'table_changes' in a notebook. However, the same query does not run in Databricks-Connect (9.1 LTS).
In Databricks-Connect I get the error below:
AnalysisException("could not resolve `table_changes` to a table-valued function; line 1
Again, the query runs fine in a notebook in a Databricks workspace.
02-08-2022 04:58 PM
Hi @Ian Corera ,
Please check the limitations from DBconnect. https://docs.databricks.com/dev-tools/databricks-connect.html#limitations It seems like the feature work fine on the notebook but not on DBconnect.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.