topic Cannot develop Delta Live Tables using Runtime 14 or 15. in Data Engineering

Cannot develop Delta Live Tables using Runtime 14 or 15.

oakhill — Fri, 23 Aug 2024 13:48:42 GMT

When trying to develop a Delta Live Table-pipeline with my very generic clusters (runtime 14.3 or 15.4 LTS), I get th following error:

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

My completely identical cluster using runtime 13.3 can import DLTs. But on Runtime 13.3 I can't improt helper modules when using Git-folders

Databricks Runtime version upgrade required Please upgrade DBR version to 14.3 or above to execute commands on notebooks inside Git folders.

I cannot find anything in the documentation to provide me with assistance on how to actually solve these problems. Any help is appreciated.

Re: Cannot develop Delta Live Tables using Runtime 14 or 15.

Brahmareddy — Fri, 23 Aug 2024 15:20:21 GMT

Hi @oakhill,

How are you doing today?

Give a try with below suggestion and let me know if it works.

To fix the DLT error, use a Managed Pipeline instead of your generic cluster, as DLT isn't supported on Runtime 14.3 or 15.4. If you need Git folder support, use Runtime 14.3+ on a separate cluster. For DLT, stick with Runtime 13.3, but handle helper modules outside Git folders. Reach out to Databricks support for more guidance if needed.

Regards,

Brahma

Re: Cannot develop Delta Live Tables using Runtime 14 or 15.

oakhill — Fri, 23 Aug 2024 20:31:57 GMT

Hi @Brahmareddy, thank you for your response. Unfortunately, attaching the notebook to a managed pipeline will cause every notebook cell to trigger a validation-update. Does not work as a developer experience at all.

Hoping someone at Databricks can chime in.

Re: Cannot develop Delta Live Tables using Runtime 14 or 15.

zoe-durand — Fri, 27 Sep 2024 21:18:54 GMT

Hi @oakhill , as stated above, in order for DLT notebooks to work well you need to create a pipeline (which it sounds like you did!).

You are correct - running a notebook cell will trigger a "Validate" action on the entire pipeline code. Alternatively, you could go to the DLT pipeline UI and trigger a "selective refresh" of one or several of your tables if you're not interested in running the entire pipeline code. https://docs.databricks.com/en/delta-live-tables/updates.html#start-a-pipeline-update-for-selected-tables

In general, you can read more about the DLT notebook development experience updates in this blog post https://www.databricks.com/blog/dlt-pipeline-development-made-simple-notebooks

Please don't hesitate to reach out if you have any more questions or feedback.

Regards,

Zoé