How to deploy a DLT Pipeline?

[jeremy98]

Hi community,

My team and I have been working on manually creating our first DLT pipeline. However, when we tried importing it into DABs, we encountered an issue in the dev workspace: we are unable to deploy the same DLT pipeline multiple times because the source is already being used by another instance of the same pipeline.

This prevents us from testing properly in the dev environment. Has anyone experienced a similar issue? How can we resolve this?

Thanks in advance!

[szymon_dybczak · Accepted solution]

Hi @jeremy98 ,

When a DLT pipeline creates a table, this table can only be written to by the pipeline that created it. If another pipeline tries to write data to this table the pipeline will error.

The solution is quite simple, parametrize your DAB template in such way that each developer during DAB deployment to DEV will have its own dedicated catalog. You can refere to below article when this approach is explained in details:

Avoid Delta Live Table Pipeline Conflicts with Parameterised Databricks Asset Bundles

[jeremy98]

Hello,

Thanks for your response! 🙂
Duplicating the catalog for this does feel a bit unusual. I understand the reasoning behind it, though it’s not the cleanest approach. Still, I suppose it’s acceptable for a DEV workspace.

Thanks again!