delta live tables - collaborative development

[ismaelhenzel]

I would like to know the best practice for collaborating on a Delta Live Tables pipeline. I was thinking that each developer should have their own DLT pipeline in the development workspace. Currently, each domain has its development catalog, like sales_dev.gold. The problem is that with DLT, I can't have two pipelines running in the same schema with the same table names. You get the error: "A table can only be owned by one pipeline. Concurrent pipeline operations such as maintenance and full refresh will conflict with each other."

So, do I need to create different pipelines for each developer, and also create different catalogs/schemas? Like sales_dev.peterson or create a catalog dev_peterson.sales? This seems a little ugly to me, but maybe I'm thinking with the wrong paradigm in mind. Since QA and Prod will have the right catalogs, maybe Dev is not so important in this organization. What do you think?

[szymon_dybczak]

Hi @ismaelhenzel ,

Yep, that's true. But you can find quite interesting workaround/solution for this problem at below excellent blog post:

https://www.advancinganalytics.co.uk/blog/avoid-delta-live-table-conflicts-with-databricks-asset-bun...

[Poorva21]

Yes—each developer should have their own DLT pipeline and their own schema. It’s the correct paradigm.
It keeps DLT ownership clean and prevents pipeline conflicts.
Dev naming doesn’t need to be pretty; QA/Prod are where structure matters.