topic Re: DLT Vs Notebook runs in Data Engineering

DLT Vs Notebook runs

aliacovella — Mon, 06 Jan 2025 20:30:22 GMT

I have this behavior that I'm not understanding. I have a notebook that defines a DLT from a Kinesis stream and a view from that DLT. This works when I run it from within workflow, configured using the DLT pipeline. If, however, I create a workflow and point it to the same notebook used in the DLT pipeline, the workflow job runs successfully but does not produce any results. Can notebooks with DLT tables not run within a workflow that points to the notebook or is there something else I need to do in order to get this to work?

Re: DLT Vs Notebook runs

hari-prasad — Tue, 07 Jan 2025 21:11:28 GMT

Hi @aliacovella ,

DLT notebooks/codes only works with DLT pipelines.

And regular Spark or SQL notebooks work with workflows.

Re: DLT Vs Notebook runs

aliacovella — Tue, 07 Jan 2025 21:28:05 GMT

Thanks, I have it working now.