cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Can't setup dbt with streaming tables

jdm
New Contributor II

Hey community,

i'm struggling integrate Delta Live Tables and dbt with one another.

Basically i'm trying to complete this tutorial. https://www.databricks.com/blog/delivering-cost-effective-data-real-time-dbt-and-databricks

Some further information:

  • Creating a delta live table with a Notebook and a Pipeline works just as expected
  • Unity Catalog is enabled for the workspace
  • Compute Cluster: Single node: Standard_DS3_v2 ยท 15.0 ML (includes Apache Spark 3.5.0, Scala 2.12)
  • SQl Warehouse: Pro, 2X-Small
1 REPLY 1

jdm
New Contributor II

i forgot to add this further discription, sorry

i added the linked github repo to my databricks workspace, successfully ran the helper notebook and created a job which runs a dbt task based on the dbt project contained in the GitHub.
This task completes with the message at the end of the post. As you can see the streaming table creations appear to be successful (e.g. 'OK created sql streaming_table model default.airline_trips_bronze' ) but can not be found in the later processing. Niether can i find them in the catalog explorer.

So in order to debug this issue i checked the "Query History" which tells me that the Create query ran successful with 1 returned row. As i do not know how else to access that output i simple pasted the query into the "SQL Editor" and executed it from there which returned one row saying: "This Delta Live Tables query is syntactically valid, but you must create a pipeline in order to define and populate your table." As shown in this [blog](https://medium.com/@kiransreekumar/streaming-tables-in-databricks-and-integration-with-dbt-08443de4b...) it at least used to be possible to run these sql statements with out a pipeline

So my questions are

1. am i missing a crucial part of the tutorial? 
2. do any of you guys face the same issue?
3. are there requirements not mentioned in the tutorial?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group