Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-11-2024 09:38 PM
Hi,
Instead of 300 individual files or one massive script, try grouping similar tables together. For example, you could have 10 scripts, each handling 30 tables. This way, you get the best of both approches—This way you will have a freedom of easy debugging without having too many files to manage.
Start with Notebooks and once everything’s running smoothly, consider converting your notebooks into .py scripts.
One more tip - look into using Delta Lake in Databricks. It makes managing your data easier and more reliable.
Give a try.