cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Cannot develop Delta Live Tables using Runtime 14 or 15.

oakhill
New Contributor II

When trying to develop a Delta Live Table-pipeline with my very generic clusters (runtime 14.3 or 15.4 LTS), I get th following error:

 

 

The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or use an existing pipeline to run DLT code. 

 

My completely identical cluster using runtime 13.3 can import DLTs. But on Runtime 13.3 I can't improt helper modules when using Git-folders

 

 

Databricks Runtime version upgrade required
Please upgrade DBR version to 14.3 or above to execute commands on notebooks inside Git folders.

 

I cannot find anything in the documentation to provide me with assistance on how to actually solve these problems. Any help is appreciated.

3 REPLIES 3

Brahmareddy
Valued Contributor II

Hi @oakhill,

How are you doing today?

Give a try with below suggestion and let me know if it works.

To fix the DLT error, use a Managed Pipeline instead of your generic cluster, as DLT isn't supported on Runtime 14.3 or 15.4. If you need Git folder support, use Runtime 14.3+ on a separate cluster. For DLT, stick with Runtime 13.3, but handle helper modules outside Git folders. Reach out to Databricks support for more guidance if needed.

Regards,

Brahma

oakhill
New Contributor II

Hi @Brahmareddy, thank you for your response. Unfortunately, attaching the notebook to a managed pipeline will cause every notebook cell to trigger a validation-update. Does not work as a developer experience at all.

Hoping someone at Databricks can chime in.

zoe-durand
New Contributor II

Hi @oakhill , as stated above, in order for DLT notebooks to work well you need to create a pipeline (which it sounds like you did!).

You are correct - running a notebook cell will trigger a "Validate" action on the entire pipeline code. Alternatively, you could go to the DLT pipeline UI and trigger a "selective refresh" of one or several of your tables if you're not interested in running the entire pipeline code. https://docs.databricks.com/en/delta-live-tables/updates.html#start-a-pipeline-update-for-selected-t...

In general, you can read more about the DLT notebook development experience updates in this blog post https://www.databricks.com/blog/dlt-pipeline-development-made-simple-notebooks

Please don't hesitate to reach out if you have any more questions or feedback.

Regards,

Zoรฉ

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group