topic Re: create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot f in Data Engineering

create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows

david_aspegren — Wed, 02 Sep 2026 09:43:25 GMT

When a pipeline contains more than one create_auto_cdc_from_snapshot_flow flow (each driven by a custom Python next_snapshot_and_version function), flow resolution fails intermittently/consistently with:

RuntimeError: The original Spark session is being accessed instead of the per-flow cloned session during parallel analysis. This is commonly caused by spawning threads inside a flow function that access the Spark session.

I am having functions to figure out the next snapshot like this:
```
def next_x_snapshot_and_version(latest_version):
versions = spark.read.table(SOURCE_TABLE).select("file_modification_time").distinct()
```

Is this a bug?

Re: create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot f

ThomasBehne — Wed, 02 Sep 2026 10:12:13 GMT

This error occurs because your custom `next_x_snapshot_and_version` function references the global `spark` session variable directly, bypassing Delta Live Tables' per-flow cloned session during parallel execution. When running multiple CDC snapshot flows concurrently, DLT isolates each flow using its own cloned session; accessing global `spark` state breaks this thread safety. To fix it, update your function signature to accept a `spark_session` argument explicitly (e.g., `def next_x_snapshot_and_version(spark_session, latest_version) and pass the thread-safe `spark` instance into your custom function via a `lambda` inside your `create_auto_cdc_from_snapshot_flow` call.