topic Re: DLT-META requires ddl when using cdc_apply_changes in Data Engineering

DLT-META requires ddl when using cdc_apply_changes

m_weirath — Fri, 16 Aug 2024 22:27:41 GMT

We are setting up new DLT Pipelines using the DLT-Meta package. Everything is going well in bringing our data in from Landing to our Bronze layer when we keep the onboarding JSON fairly vanilla. However, we are hitting an issue when using the cdc_apply_changes feature. As soon as we add that feature, the package will look for a .ddl schema file. Since we aren't using one, the package will fail when we start the pipelines. Complaining it can't parse a null file

For a bit of background, we plan on using schema evolution and would like to use the cdc_apply_changes to keep the bronze tables mostly similar to the source systems.

The cdc_apply_changes doesn't need a schema file, so I am confused about why one is now required. Is this just a bug in the DLT-Meta package? Does anyone have any suggestions?

Re: DLT-META requires ddl when using cdc_apply_changes

Tanay — Sat, 17 Aug 2024 03:45:10 GMT

You're hitting a bug or an unexpected behavior in the DLT-Meta package. Normally, the <code>cdc_apply_changes</code> feature shouldn't require a <code>.ddl</code> schema file, especially if you're relying on schema evolution.

<ul>
<li>Check the package version: Ensure you're using the latest version of the DLT-Meta package. Sometimes, bugs are fixed in newer releases.</li>
<li>Look for a workaround: If the schema file is being incorrectly required, see if there's a way to bypass this by providing a minimal or empty schema file as a temporary fix.</li>
<li>Reach out to support: If it seems like a bug, reporting it to the maintainers of the DLT-Meta package might help. They might already have a fix or workaround.</li>
</ul>

In the meantime, consider testing with a simple schema file to keep your pipeline moving forward.

Re: DLT-META requires ddl when using cdc_apply_changes

dbuser17 — Tue, 03 Sep 2024 03:33:42 GMT

Please check these details: https://github.com/databrickslabs/dlt-meta/issues/90