cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Notebook and Pipeline Separation vs Consolidation

ChristianRRL
Contributor II

Super basic question. For DLT pipelines I see there's an option to add multiple "Paths". Is it generally best practice to completely separate `bronze` from `silver` notebooks? Or is it more recommended to bundle both raw `bronze` and clean `silver` data into notebooks together and separate notebooks by something else?

Similarly, what is an "appropriate" number of tables to process in a single DLT pipeline? E.g. I'm already processing 20 tables from one source and because they're all bronze I have to squint (or realistically just zoom in on the DLT Graph). These 20 bronze tables would flow into 20 silver tables for this current example, but for separate notebook (+maybe separate DLT pipeline) I could see 100s of bronze tables flowing into a few distinct silver tables.

ChristianRRL_1-1705597040187.png

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Esteemed Contributor
Esteemed Contributor

You can run as many tables as you want provided the cluster capacity. Also, if you are processing large no. of tables, using list view in DLT might be a better option as compared to graph view.

View solution in original post

2 REPLIES 2

Lakshay
Esteemed Contributor
Esteemed Contributor

You can run as many tables as you want provided the cluster capacity. Also, if you are processing large no. of tables, using list view in DLT might be a better option as compared to graph view.

ChristianRRL
Contributor II

This is great! I completely missed the list view before.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!