Can't setup dbt with streaming tables

jdm — Thu, 25 Apr 2024 12:48:46 GMT

Hey community,

i'm struggling integrate Delta Live Tables and dbt with one another.

Basically i'm trying to complete this tutorial. https://www.databricks.com/blog/delivering-cost-effective-data-real-time-dbt-and-databricks

Some further information:

Creating a delta live table with a Notebook and a Pipeline works just as expected
Unity Catalog is enabled for the workspace
Compute Cluster: Single node: Standard_DS3_v2 · 15.0 ML (includes Apache Spark 3.5.0, Scala 2.12)
SQl Warehouse: Pro, 2X-Small

Re: Can't setup dbt with streaming tables

jdm — Thu, 25 Apr 2024 12:56:29 GMT

i forgot to add this further discription, sorry

i added the linked github repo to my databricks workspace, successfully ran the helper notebook and created a job which runs a dbt task based on the dbt project contained in the GitHub.
This task completes with the message at the end of the post. As you can see the streaming table creations appear to be successful (e.g. 'OK created sql streaming_table model default.airline_trips_bronze' ) but can not be found in the later processing. Niether can i find them in the catalog explorer.

So in order to debug this issue i checked the "Query History" which tells me that the Create query ran successful with 1 returned row. As i do not know how else to access that output i simple pasted the query into the "SQL Editor" and executed it from there which returned one row saying: "This Delta Live Tables query is syntactically valid, but you must create a pipeline in order to define and populate your table." As shown in this [blog](https://medium.com/@kiransreekumar/streaming-tables-in-databricks-and-integration-with-dbt-08443de4bd44) it at least used to be possible to run these sql statements with out a pipeline

So my questions are

1. am i missing a crucial part of the tutorial?
2. do any of you guys face the same issue?
3. are there requirements not mentioned in the tutorial?

topic Can't setup dbt with streaming tables in Get Started Discussions

Can't setup dbt with streaming tables

Re: Can't setup dbt with streaming tables