dbt+Databrics

[Luke_Kociuba]

Hi!
I will use dbt + Databrics in my new project.
I'm runing dbt training. Have Databrics free/trial account so far. Want to connect, link dbt with Databbrics and create data warehouse from below data sets
jaffle_shop_customers.csv
jaffle_shop_orders.csv
stripe_payments.csv

How to achieve it?

Create SQL warehouse button is inactive (greyed) and cannot see the "Manage entitlements" option under "Worspace" and do not see "Allow unrestricted cluster creation" button/permision required to create a SQL warehouse.

Can you give me some hints how to proceed with the dbt/Databrics traininig?

Kind regards,

Luke

[szymon_dybczak · Accepted solution]

Hi @Luke_Kociuba ,

I'm assuming you're using databricks free edtion. In free edition you can't create new SQL Warehouse, but you can use one that is already created for you called called Serverless Started Warehouse (just like I show on the screen below):

So, first step is to create some tables from the sample data they provided. To do so:

1. click New button (1) and then Add or Upload Data (2): 

2. Next, click create or modify table

3. Drag and drop your CSV file (i.e jaffle_shop_customers.csv)

4. Click create table

5. Perform above steps for 2 other files 

•  jaffle_shop_orders.csv
• stripe_payments.csv

6. Now you should able to see new table in a workspace catalag within default schema

Now, you need to configure dbt connecton. To do so perform following steps:

1. Add dbt connection

2. Choose databricks

2. Click on SQL Warehouse button on the left hand-sight of UI and then click on the Serveless Starter Warehouse:

3. Next, Click connection details

4. Copy your connection details - in the next step we paste them to DBT config screen

4. Paste above connection details so we can configure DBT connection to our warehouse. In the optional settings provide catalog name = "Workspace"

5. Now, return to the New project wizard and select the connection you just created from the Connection drop-down menu. We need to provide an access token (refer to next step how to acquire access token)

5.1 To acquire access token in databricks click on you initials button in the upper right corner of the screen and click setting:

5.2 Now click developer (1) and then manage (2)

5.3. Click generate new access token and then generate. Copy the generated access token and paste that token to dbt setting page (step 5).

6. After you connect to a Databricks cluster or a Databricks SQL warehouse, follow the on-screen instructions to Setup a Repository, and then click Continue.

7. And that should be it, from here follow your tutorial

[BS_THE_ANALYST]

Great solution @szymon_dybczak 👏🙂.

I don't know too much about DBT yet. Is there much difference between connecting/interacting with DBT cloud vs DBT core?

All the best,
BS

[szymon_dybczak]

Hi @BS_THE_ANALYST ,

Thanks 🙂 There's not that much difference when it comes to configuring DBT core. But if you're going to learn DBT you can create DBT cloud account - it's free for learning purposes 🙂 I think for 14 days you have an access to 14 days trial and then you can still use it for free - but you have a limitation of only 1 project