Unity Catalog Not Enabled on Job Cluster When Creating DLT in GCP Databricks

[vidya_kothavale]

I am trying to create a Delta Live Table (DLT) in my GCP Databricks workspace, but I am encountering an issue where Unity Catalog is not enabled on the job cluster.

Steps I followed:

1. Created a DLT pipeline using the Databricks UI.
2. Selected the appropriate Compute settings for the pipeline.
3. Configured the pipeline to use Unity Catalog.
4. Attempted to run the pipeline, but got the error:

   Unity Catalog is not enabled on this job cluster

Observations:

• My cluster is running Databricks Runtime dlt:15.4.8
• The Access Mode is set to Standard (formerly Shared)
• The worker type is e2-standard-8 (32 GB Memory, 8 Cores)

Troubleshooting did:

Verified that Unity Catalog is enabled in the workspace.
Checked if the correct metastore is assigned to the workspace.
Ensured that the cluster is configured with the right IAM roles and policies.

Additional Finding:

• If I create a standard Databricks job, Unity Catalog is enabled on the job compute.
• The issue occurs only when I create a Delta Live Table (DLT) pipeline, where Unity Catalog is not enabled on the job cluster.