Databricks Community

@MauricioS Great question!Databricks Delta Live Tables (DLT) pipelines are very flexible, but by default, the target schema specified in the pipeline configuration (such as target or schema) is fixed. That said, you can implement strategies to enable...

Hi @Alberto_Umana , Thanks for replying.Even if we add the driver node it should be around 16-17 right, not like 20. I checked for al the clusters, for every cluster there is a difference of 5-7 nodes between max_worker count and count(distinct insta...

Unfortunately the system events are only tracked via the system table, only option to have more recent data will be to re execute the query each time is needed.

Unfortunately there is no ETA on this yet. If I know about it, will let you know!

I would suggest you to submit this as a feature request through https://docs.databricks.com/en/resources/ideas.html#ideas 

Was this solved @Rajeev45  Do you have any docs?

Hi @sahasimran98 I think you're right this is more valid for synapse where such configuration exist but you can still give a try for databricks and let us know here the results. otherwise try to find some spark-monitoring package in github for databr...

you can make use of databricks native feature "Liquid Clustering", cluster by the columns which you want to use in grouping statements, it will handle the performance issue due to data skewness .For more information, please do visit :https://docs.dat...

I do did the similar kind of work in my recent project, where I need to run many SQL DDL's , so I automated the process using databricks jobs, capturing the dependency using a metadata table and creating tasks likewise in job through job api's, doing...

@adityarai316  did u get the solution because I am facing the same issue?