topic Dlt pipeline showing legacy , even though all things are latest in Data Engineering

Dlt pipeline showing legacy , even though all things are latest

21f3001806 — Wed, 16 Apr 2025 05:37:28 GMT

Some of old dlt pipelines in my databricks workspace are showing legacy,

I am using serverless pipeline with mode - preview.

Anything which I missed ?

Re: Dlt pipeline showing legacy , even though all things are latest

RiyazAliM — Wed, 16 Apr 2025 08:50:34 GMT

Hello @21f3001806

Would you be able to share the JSON of your pipeline settings, having no info on your pipeline makes difficult to determine what's going on.

Please make sure to mask any PII info in the JSON before sharing.

Cheers!

Re: Dlt pipeline showing legacy , even though all things are latest

ashraf1395 — Mon, 21 Apr 2025 03:53:16 GMT

Hi @21f3001806 , I recently got a similar error for my old pipelines. so databricks recently made dlt direct publising mode ga which caused some changes in the dlt pipeline configuration definition so if you use the setting 'target' in your pipeline it will show legacy , if you change it to 'schema' it will be corrected. Basically it means that dlt supporting unity catalog.

here's the link : https://www.databricks.com/blog/publish-multiple-catalogs-and-schemas-single-dlt-pipeline

Not sure whether this is the right solution but you can give it a try

Re: Dlt pipeline showing legacy , even though all things are latest

RiyazAliM — Mon, 21 Apr 2025 06:53:28 GMT

Hi @ashraf1395 - I'm trying to reproduce the `legacy` pipeline with no luck, I'm trying to understand - so basically, I should select the destination as Hive metastore instead of UC?

Thanks for the helpful link with updates.

Re: Dlt pipeline showing legacy , even though all things are latest

ashraf1395 — Mon, 21 Apr 2025 10:00:26 GMT

Hi @RiyazAliM , I am not sure whether that works or not.
In my case :
the dlt pipeline configuration : instead of use target as a parameter instead of schema
Like in this below image it show legacy

Re: Dlt pipeline showing legacy , even though all things are latest

RiyazAliM — Thu, 24 Apr 2025 10:57:13 GMT

@ashraf1395 - I understand now, let me try it once.