Databricks Community

smoortema

I am working on a testing notebook where the table that is tested can be given as a widget. I wanted to write it in SQL. The notebook does the following steps in a cycle that should run 10 times:1. Store the starting version of a delta table in a var...

smoortema

In a Python notebook, I am using error handling according to the official documentation. try:[some data transformation steps]except PySparkException as ex:[logging steps to log the error condition and error message in a table]However, this catches o...

smoortema · 07-21-2025

We need to set up a job that consists of several hundred tasks with many dependencies between each other. We are considering two different directions:1. Databricks job with tasks, with dependencies defined as code and deployed with Databricks asset b...

smoortema

Great, thanks, I was not familiar with stacking exception handlers from most specific to most general, so I had two try rows in the beginning instead of only one.

smoortema · 07-22-2025

I have a similar question, but I am more interested in it to understand whether resources are allocated in an intelligent way. I guess the compute configuration should have an effect on how many tasks can be run at the same time. But how does it exac...

Databricks Community

User Stats

User Activity

How to make FOR cycle and dynamic SQL and variables work together

handling both Pyspark and Python exceptions

Complex pipeline with many tasks and dependencies: orchestration in Jobs or in Notebook?

Re: handling both Pyspark and Python exceptions

Re: Limiting concurrent tasks in a job