Is Delta Live Tables not supported anymore? How do I use it in Python?

[oakhill]

Hi!

Any time I try to import "dlt" in a notebook session to develop Pipelines, I get an error message saying DLT is not supported on Spark Connect clusters. These are very generic clusters, I've tried runtime 14, 15 and the latest 16, using shared clusters as recommended.

How are we supposed to use DLT?

[BigRoux]

Hey Oakhill, are you trying to create a DLT pipeline outside of the Databricks environment? Provide more details of your setup. Thanks, Louis.

[oakhill]

Hi,

I am sitting in my Databricks developent workspace. We've mostly used Structured Streaming but wanted to try DLT and Autoloader.
I create a very generic cluster using the LTS spark version, VM is Standard_D4s_v3 and Access Mode is shared.

See screenshot for error.

[BigRoux]

Delta Live tables does not support shared clusters.  You need to configure the cluster (a jobs cluster) via the Delta Live Tables use (under workflows).  You would still write your notebook with DLT using Python but the pipeline itself has to be configured via the UI and run from the UI.

[oakhill]

I am aware that I have to specify a job cluster/pipeline when deploying. But I am asking how to actually develop on a newer DBR. Back in 12 and 13 we would share a cheap development cluster and then push to prod with autoscale and larger clusters.

I just booted up a cluster on DBR 13 and it worked. Why is that? It's an old version and doesn't support Liquid Clustering, for example.

EDIT: OK, so I had to create a user isolated cluster on DBR 15, unlike the shared on DBR 13.That's unfortunate that we have to make individual clusters for DLT.

[Fabricio_Mattos]

Hey Oakhill,

Are trying to executed the notebook clicking the run button? If is the case, it not gonna work. You need to write the all the code first. After that You need to go in the sidebar menu click in workflow, choose the table Delta Live table and click in the blue button in the right corner to create a pipeline. 

Then you need to setup your pipeline with information about you cluster, destination, the path of your notebook. After that , click in create, and your pipeline will be create. Then click in the start button to start you DLT pipeline.

[oakhill]

No, this is when I try to develop on a DBR above 13. I can't even import the dlt-package when developing 🙂 

[BigRoux]

Oakhill. When building a DLT pipeline you are not prompted for DBR when defining the cluster.    DLT handles the infrastructure rollout for you.  Can you please share a screenshot of the error you are recieving, I want to see where you are in the platform when you get the error.

[oakhill]

Hi BigRoux, thanks for the patience.

When developing, I use a standard notebook and attach it to a running cluster. This is not when I _deploy_ the DLT.
This is when the error occurs.

Surely we are not supposed to develop when running the actual pipeline using the validate button? Do you normally develop on an active pipeline?

I have added a screenshot.

[BigRoux]

Oakhill, we do provide free onboard training. You might be interested in the "Get Started with Data Engineering on Databricks" session.  You can register here: https://www.databricks.com/training/catalog.  When you are searching the catalog of training be sure to filter on "Type" for "Get Started." Cheers, Louis.