cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Can't setup dbt with streaming tables

jdm
New Contributor II

Hey community,

i'm struggling integrate Delta Live Tables and dbt with one another.

Basically i'm trying to complete this tutorial. https://www.databricks.com/blog/delivering-cost-effective-data-real-time-dbt-and-databricks

Some further information:

  • Creating a delta live table with a Notebook and a Pipeline works just as expected
  • Unity Catalog is enabled for the workspace
  • Compute Cluster: Single node: Standard_DS3_v2 ยท 15.0 ML (includes Apache Spark 3.5.0, Scala 2.12)
  • SQl Warehouse: Pro, 2X-Small
1 REPLY 1

jdm
New Contributor II

i forgot to add this further discription, sorry

i added the linked github repo to my databricks workspace, successfully ran the helper notebook and created a job which runs a dbt task based on the dbt project contained in the GitHub.
This task completes with the message at the end of the post. As you can see the streaming table creations appear to be successful (e.g. 'OK created sql streaming_table model default.airline_trips_bronze' ) but can not be found in the later processing. Niether can i find them in the catalog explorer.

So in order to debug this issue i checked the "Query History" which tells me that the Create query ran successful with 1 returned row. As i do not know how else to access that output i simple pasted the query into the "SQL Editor" and executed it from there which returned one row saying: "This Delta Live Tables query is syntactically valid, but you must create a pipeline in order to define and populate your table." As shown in this [blog](https://medium.com/@kiransreekumar/streaming-tables-in-databricks-and-integration-with-dbt-08443de4b...) it at least used to be possible to run these sql statements with out a pipeline

So my questions are

1. am i missing a crucial part of the tutorial? 
2. do any of you guys face the same issue?
3. are there requirements not mentioned in the tutorial?