DLT Flow Failed Due to Missing Flow Checkpoints Directory When Using Unity Catalog

[minhhung0507]

I’m encountering an issue while running a Delta Live Tables (DLT) pipeline that is managed using Unity Catalog on Databricks. The pipeline has failed and is not restarting, showing the following error:

java.lang.IllegalArgumentException: flow checkpoints directory is not defined. Please contact Databricks support.
Stack trace snippet:

at com.databricks.pipelines.execution.core.FlowSystemMetadata.$anonfun$flowCheckpointsDir$1...
at com.databricks.pipelines.execution.core.FlowPlanner.plan...
at com.databricks.pipelines.execution.core.GraphExecution.$anonfun$startFlow$1...
...

Context:

• I'm managing the DLT pipeline with Unity Catalog.

• The flow that failed is: uat4_gold.lakehouse.gold_customerinfo

• I'm using the new Databricks UI for monitoring pipeline runs.

• The error says that the "flow checkpoints directory is not defined", but I didn’t explicitly set any custom checkpoint directory (expecting defaults to work).

• This issue prevents the pipeline from even starting execution.

My Questions:

1. What is the root cause of this flow checkpoints directory is not defined error in the context of Unity Catalog and DLT?

2. How can I correctly configure the checkpoint location for DLT flows under Unity Catalog? Is there a setting I might be missing?

3. Is this a known issue with Unity Catalog and DLT pipelines (possibly a bug or configuration oversight)?

4. What are the best practices for setting up checkpointing in DLT pipelines when using Unity Catalog?

Any help or pointers (including relevant documentation) would be greatly appreciated.

Thanks in advance!

[minhhung0507]