Add custom tags to DLT cluster

[BradSheridan]

We have a policy in our AWS account that whenever an EC2 instance is created, there are 5 mandatory tags that need to be added. When we create an All-Purpose cluster in the Databricks console, we can easily add these under Advance Options > Tags. However, this isn't available when creating a DLT job under Workflows > Delta Live Tables. Now the EC2 instances that are supposed to spin up for the DLT job never get created b/c they don't have tags. When we go to Compute and Terminate the cluster creation to edit it and add tags, we get a message "Error: dlt prefixed spark images cannot be used outside of the Delta Live Tables service". Any help is greatly appreciated! Brad

[tomasz]

You can add "custom_tags" in the DLT configuration under "clusters" (in JSON view):

 

The full range of attributes allowed in this section are listed here.

[BradSheridan]

@Tomasz Bacewicz​ Perfect!! worked like a charm...thank you!!

do you think auto loader or DLT is a better approach to implementing a CDC pipeline in Databricks?

[tomasz · Accepted solution]

Great!

When it comes to your question, autoloader and DLT can be used together so they are not mutually exclusive, with autoloader being the mechanism to load data into your DLT pipeline. It is common for both of them to be used when implementing a CDC pipeline.

[BradSheridan]

Thanks again!