The best practices for setting up checkpointing in Delta Live Tables (DLT) pipelines when using Unity Catalog are largely centered on leveraging Databricks' managed services, adhering to Unity Catalog's table management conventions, and minimizing the need for manual checkpoint directory configuration. DLT and Unity Catalog work together to abstract away most of the complexity around checkpoint storage, provided common configuration principles are observed.​

Use Managed Tables and Default Locations

• DLT pipelines with Unity Catalog always use managed tables; Databricks internally manages the checkpoint and metadata directories in the managed storage location associated with the catalog and schema.​

• Manual configuration of checkpoint locations is not required or generally supported. Rely on the defaults unless there is a specialized need.

Unity Catalog & Metadata Storage Best Practices

• Always deploy DLT pipelines into Unity Catalog schemas that have a valid managed location defined in your cloud storage provider.​

• Do not attempt to specify custom checkpoint paths or table locations, as this can lead to errors or lack of support in Unity Catalog-enabled pipelines.​

• Use “describe extended” or Unity Catalog's catalog viewer to inspect the actual storage path if needed.​

Permissions and Access Control

• Ensure that all compute clusters and resources running the pipeline have appropriate access to the managed storage location defined for the target schema.​

• Confirm that users and service principals have sufficient privileges to create/modify tables and materialized views in the target Unity Catalog locations.​

Streaming Table & Checkpoint Handling

• For streaming tables, DLT internally manages checkpointing to track incremental progress, ensuring exactly-once processing semantics. No extra setup is necessary for checkpointing, even in triggered mode.​

• For additional raw file ingestion or hybrid scenarios, create volumes within the same Unity Catalog schema for ingesting/handling files, but keep DLT datasets themselves managed by the platform.​

• If you need to manually handle checkpoints for external streaming sources, use Unity Catalog Volumes for checkpoint directory paths, but this is only for advanced cases.​

Cleanup and Lifecycle

• Lifecycle of checkpoint folders is managed automatically when you drop a table or modify pipeline definitions. If you use any volumes for streaming checkpoints outside DLT tables, manual cleanup may be necessary when dropping those tables.