Databricks Community

oakhill

Hi!Any time I try to import "dlt" in a notebook session to develop Pipelines, I get an error message saying DLT is not supported on Spark Connect clusters. These are very generic clusters, I've tried runtime 14, 15 and the latest 16, using shared clu...

oakhill

Hi!I am struggling to understand how to properly manage my table to make queries effective. My table has columns date_time_utc, car_id, car_owner etc. date_time_utc, car_id and position is usually the ZORDER or Liquid Clustering-columns.Selecting max...

oakhill · 08-23-2024

When trying to develop a Delta Live Table-pipeline with my very generic clusters (runtime 14.3 or 15.4 LTS), I get th following error: The Delta Live Tables (DLT) module is not supported on this cluster. You should either create a new pipeline or us...

oakhill · 07-02-2024

We receive several datasets where the full dump is delivered daily or weekly. What is the best way to ingest this into Databricks using DLT or basic PySpark while adhering to the medallion?1. If we use AutoLoader into Bronze, We'd end up with increme...

oakhill

No, this is when I try to develop on a DBR above 13. I can't even import the dlt-package when developing 

oakhill

I am aware that I have to specify a job cluster/pipeline when deploying. But I am asking how to actually develop on a newer DBR. Back in 12 and 13 we would share a cheap development cluster and then push to prod with autoscale and larger clusters.I ...

oakhill

Hi,I am sitting in my Databricks developent workspace. We've mostly used Structured Streaming but wanted to try DLT and Autoloader.I create a very generic cluster using the LTS spark version, VM is Standard_D4s_v3 and Access Mode is shared.See screen...

oakhill · 08-23-2024

Hi @Brahmareddy, thank you for your response. Unfortunately, attaching the notebook to a managed pipeline will cause every notebook cell to trigger a validation-update. Does not work as a developer experience at all.Hoping someone at Databricks can c...

oakhill · 07-02-2024

Won't this cause troubles with CDC in the silver layer because the entire dataset is new? Or will it remember what lines from the bronze it already has read even though it's overwritten?

Databricks Community

User Stats

User Activity

Is Delta Live Tables not supported anymore? How do I use it in Python?

How to optimize queries on a 150B table? ZORDER, LC or partioning?

Cannot develop Delta Live Tables using Runtime 14 or 15.

Optimal process for loading data where the full dataset is provided every day?

Re: Is Delta Live Tables not supported anymore? How do I use it in Python?

Re: Is Delta Live Tables not supported anymore? How do I use it in Python?

Re: Is Delta Live Tables not supported anymore? How do I use it in Python?

Re: Cannot develop Delta Live Tables using Runtime 14 or 15.

Re: Optimal process for loading data where the full dataset is provided every day?