Cannot develop Delta Live Tables using Runtime 14 or 15.

oakhill
New Contributor III

‎08-23-2024 06:48 AM

When trying to develop a Delta Live Table-pipeline with my very generic clusters (runtime 14.3 or 15.4 LTS), I get th following error:

 

 

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code.

 

My completely identical cluster using runtime 13.3 can import DLTs. But on Runtime 13.3 I can't improt helper modules when using Git-folders

 

 

Databricks Runtime version upgrade required
Please upgrade DBR version to 14.3 or above to execute commands on notebooks inside Git folders.

 

I cannot find anything in the documentation to provide me with assistance on how to actually solve these problems. Any help is appreciated.

0 Kudos

Brahmareddy
Esteemed Contributor II

‎08-23-2024 08:20 AM

Hi @oakhill,

How are you doing today?

Give a try with below suggestion and let me know if it works.

To fix the DLT error, use a Managed Pipeline instead of your generic cluster, as DLT isn't supported on Runtime 14.3 or 15.4. If you need Git folder support, use Runtime 14.3+ on a separate cluster. For DLT, stick with Runtime 13.3, but handle helper modules outside Git folders. Reach out to Databricks support for more guidance if needed.

Regards,

Brahma

0 Kudos

oakhill
New Contributor III

‎08-23-2024 01:31 PM

Hi @Brahmareddy, thank you for your response. Unfortunately, attaching the notebook to a managed pipeline will cause every notebook cell to trigger a validation-update. Does not work as a developer experience at all.

Hoping someone at Databricks can chime in.

0 Kudos

zoe-durand
Databricks Employee
Databricks Employee

‎09-27-2024 02:18 PM

Hi @oakhill , as stated above, in order for DLT notebooks to work well you need to create a pipeline (which it sounds like you did!).

You are correct - running a notebook cell will trigger a "Validate" action on the entire pipeline code. Alternatively, you could go to the DLT pipeline UI and trigger a "selective refresh" of one or several of your tables if you're not interested in running the entire pipeline code. https://docs.databricks.com/en/delta-live-tables/updates.html#start-a-pipeline-update-for-selected-t...

In general, you can read more about the DLT notebook development experience updates in this blog post https://www.databricks.com/blog/dlt-pipeline-development-made-simple-notebooks

Please don't hesitate to reach out if you have any more questions or feedback.

Regards,

Zoé

0 Kudos