Error deploying DLT DAB: Validation failed for cluster_type, the value must be dlt (is "job")

[ahab]

Hello.

I'm getting a cluster validation error while trying to deploy a DLT pipeline via DAB.

See attached screenshots for config and error.

Hoping someone has run into this before and can guide me.

Thanks.

[Kaniz]

Hi @ahab, Deploying a Delta Live Tables (DLT) pipeline via Databricks can sometimes be tricky, especially when dealing with cluster validation errors. 

Let’s troubleshoot this together.

Cluster Validation Error:

• When encountering a cluster validation error during DLT pipeline deployment, consider the following steps:
  • Check Data Updates: Verify if there are any unexpected data updates in your source table. Ensure that the data remains consistent during pipeline execution1.
  • Resource Quotas: Sometimes, the issue might be related to resource quotas. Ensure that you have sufficient compute resources available. If you’re using a specific vCPU type, make sure it’s not at its quota. You can explicitly specify a vCPU type in your settings.json configuration for the pipeline2.

Pipeline Settings:

• Product Edition: Select the appropriate DLT product edition based on your pipeline requirements:
  • Core: For streaming ingest workloads without advanced features like change data capture (CDC) or DLT expectations.
  • Pro: Supports streaming ingest and CDC workloads, including updating tables based on changes in source data.
  • Advanced: Includes Core and Pro features, plus support for enforcing data quality constraints using DLT expectations.
• Pipeline Mode: Choose between continuous execution or manual triggers based on your needs3.

Remember that DLT settings can be configured via the UI or a JSON specification. For advanced options, explore the JSON configuration. If you encounter any issues, feel free to adjust the product edition or other settings as needed. Happy troubleshooting! 🚀