cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Notebook and Pipeline Separation vs Consolidation

ChristianRRL
Contributor III

Super basic question. For DLT pipelines I see there's an option to add multiple "Paths". Is it generally best practice to completely separate `bronze` from `silver` notebooks? Or is it more recommended to bundle both raw `bronze` and clean `silver` data into notebooks together and separate notebooks by something else?

Similarly, what is an "appropriate" number of tables to process in a single DLT pipeline? E.g. I'm already processing 20 tables from one source and because they're all bronze I have to squint (or realistically just zoom in on the DLT Graph). These 20 bronze tables would flow into 20 silver tables for this current example, but for separate notebook (+maybe separate DLT pipeline) I could see 100s of bronze tables flowing into a few distinct silver tables.

ChristianRRL_1-1705597040187.png

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Esteemed Contributor

You can run as many tables as you want provided the cluster capacity. Also, if you are processing large no. of tables, using list view in DLT might be a better option as compared to graph view.

View solution in original post

2 REPLIES 2

Lakshay
Esteemed Contributor

You can run as many tables as you want provided the cluster capacity. Also, if you are processing large no. of tables, using list view in DLT might be a better option as compared to graph view.

ChristianRRL
Contributor III

This is great! I completely missed the list view before.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group