topic Re: Can we use %run or any magic commands in a Delta Live Table pipeline? in Data Engineering

Can we use %run or any magic commands in a Delta Live Table pipeline?

Manimeghala — Mon, 12 Jun 2023 09:18:53 GMT

When we try, it says magic commands are not supported. Is there a way to import other notebooks first so that its functions can be referenced/used while we build a DLT pipeline?

Re: Can we use %run or any magic commands in a Delta Live Table pipeline?

Manimeghala — Mon, 12 Jun 2023 12:34:54 GMT

Thanks @Kaniz Fatma

As per the above article, %run is not supported for DLT

Do we have any alternate solution for this rather than combining my utils notebook and main notebook

Re: Can we use %run or any magic commands in a Delta Live Table pipeline?

-werners- — Tue, 13 Jun 2023 13:29:59 GMT

perhaps you can refactor your utils notebook to python modules (so .py files)

https://learn.microsoft.com/en-us/azure/databricks/delta-live-tables/import-workspace-files

Re: Can we use %run or any magic commands in a Delta Live Table pipeline?

Anonymous — Wed, 14 Jun 2023 03:21:18 GMT

Hi @Manimeghala Vemula

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Can we use %run or any magic commands in a Delta Live Table pipeline?

Manimeghala — Wed, 14 Jun 2023 05:56:34 GMT

@Werner Stinckens Thanks a ton for your response, Converting notebooks to python modules did help us in the current use-case. We are able to import.