topic Re: Add custom tags to DLT cluster in Data Engineering

Add custom tags to DLT cluster

BradSheridan — Mon, 11 Jul 2022 12:24:08 GMT

We have a policy in our AWS account that whenever an EC2 instance is created, there are 5 mandatory tags that need to be added. When we create an All-Purpose cluster in the Databricks console, we can easily add these under Advance Options > Tags. However, this isn't available when creating a DLT job under Workflows > Delta Live Tables. Now the EC2 instances that are supposed to spin up for the DLT job never get created b/c they don't have tags. When we go to Compute and Terminate the cluster creation to edit it and add tags, we get a message "Error: dlt prefixed spark images cannot be used outside of the Delta Live Tables service". Any help is greatly appreciated! Brad

Re: Add custom tags to DLT cluster

tomasz — Mon, 11 Jul 2022 13:44:46 GMT

You can add "custom_tags" in the DLT configuration under "clusters" (in JSON view):

The full range of attributes allowed in this section are listed here.

Re: Add custom tags to DLT cluster

BradSheridan — Mon, 11 Jul 2022 18:00:57 GMT

@Tomasz Bacewicz Perfect!! worked like a charm...thank you!!

do you think auto loader or DLT is a better approach to implementing a CDC pipeline in Databricks?

Re: Add custom tags to DLT cluster

tomasz — Mon, 11 Jul 2022 18:48:08 GMT

Great!

When it comes to your question, autoloader and DLT can be used together so they are not mutually exclusive, with autoloader being the mechanism to load data into your DLT pipeline. It is common for both of them to be used when implementing a CDC pipeline.

Re: Add custom tags to DLT cluster

BradSheridan — Mon, 11 Jul 2022 19:18:40 GMT

Thanks again!