cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Discussions
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Can't setup dbt with streaming tables

jdm
New Contributor II

Hey community,

i'm struggling integrate Delta Live Tables and dbt with one another.

Basically i'm trying to complete this tutorial. https://www.databricks.com/blog/delivering-cost-effective-data-real-time-dbt-and-databricks

Some further information:

  • Creating a delta live table with a Notebook and a Pipeline works just as expected
  • Unity Catalog is enabled for the workspace
  • Compute Cluster: Single node: Standard_DS3_v2 ยท 15.0 ML (includes Apache Spark 3.5.0, Scala 2.12)
  • SQl Warehouse: Pro, 2X-Small
1 REPLY 1

jdm
New Contributor II

i forgot to add this further discription, sorry

i added the linked github repo to my databricks workspace, successfully ran the helper notebook and created a job which runs a dbt task based on the dbt project contained in the GitHub.
This task completes with the message at the end of the post. As you can see the streaming table creations appear to be successful (e.g. 'OK created sql streaming_table model default.airline_trips_bronze' ) but can not be found in the later processing. Niether can i find them in the catalog explorer.

So in order to debug this issue i checked the "Query History" which tells me that the Create query ran successful with 1 returned row. As i do not know how else to access that output i simple pasted the query into the "SQL Editor" and executed it from there which returned one row saying: "This Delta Live Tables query is syntactically valid, but you must create a pipeline in order to define and populate your table." As shown in this [blog](https://medium.com/@kiransreekumar/streaming-tables-in-databricks-and-integration-with-dbt-08443de4b...) it at least used to be possible to run these sql statements with out a pipeline

So my questions are

1. am i missing a crucial part of the tutorial? 
2. do any of you guys face the same issue?
3. are there requirements not mentioned in the tutorial?

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.