topic Databricks-Connect and Change Data Feed query error in Data Engineering

Databricks-Connect and Change Data Feed query error

Ian — Mon, 03 Jan 2022 18:49:20 GMT

I have installed Databricks-Connect (9.1 LTS). I am able to send queries to the cluster. However, when the query includes a call to the 'table_changes' function that is a part of Change Data Feed, I get the following error:

AnalysisException("could not resolve `table_changes` to a table-valued function; line 1

The query runs when I run it in a notebook against the same cluster. The query looks similar to the following:

select distinct SomeFieldID from table_changes('SomeTable','2021-12-31 9:53:02') where _change_type in ('update_postimage')

Re: Databricks-Connect and Change Data Feed query error

Anonymous — Mon, 03 Jan 2022 23:17:53 GMT

Hello @Ian Corera - My name is Piper, and I'm one of the moderators for Databricks. Let's see what your fellow members have to say before we circle back around to this. 🙂

Re: Databricks-Connect and Change Data Feed query error

Ian — Fri, 21 Jan 2022 19:00:36 GMT

Hi @Kaniz Fatma , the table_changes function is an internal Databricks function used in Change Data Feed (CDF).

Please refer to the article below. It discusses the table_changes function.

https://docs.databricks.com/delta/delta-change-data-feed.html

Re: Databricks-Connect and Change Data Feed query error

Ian — Mon, 24 Jan 2022 13:44:26 GMT

Hi @Kaniz Fatma thank you for your reply. I do have the tables setup for Change Data Feed. I am able to run the query with 'table_changes' in a notebook. However, the same query does not run in Databricks-Connect (9.1 LTS).

In Databricks-Connect I get the error below:

AnalysisException("could not resolve `table_changes` to a table-valued function; line 1

Again, the query runs fine in a notebook in a Databricks workspace.

Re: Databricks-Connect and Change Data Feed query error

jose_gonzalez — Wed, 09 Feb 2022 00:58:07 GMT

Hi @Ian Corera ,

Please check the limitations from DBconnect. https://docs.databricks.com/dev-tools/databricks-connect.html#limitations It seems like the feature work fine on the notebook but not on DBconnect.