Databricks Community

smoortema

I have a Delta table on which I set up liquid clustering using three columns. I would like to check file statistics to see how the clustering column values are distributed along the files. How can I write a query that shows min and max values, etc. o...

smoortema · 02-16-2026

I want to build a job or a set of jobs that load data from several different source systems (with approximately 600 tasks in total). Data from the source systems arrive at different points in time every day, and a query needs to be executed to see if...

smoortema · 02-05-2026

I have a notebook that I am able to run in itself, but when I run it within a Databricks job, it fails with the following error:[UNAUTHORIZED_ACCESS] PERMISSION_DENIED: User does not have MANAGE on Connection 'dev_postgres'. SQLSTATE: 42501The notebo...

smoortema · 11-18-2025

What is the best way to know what kind of join was used for a SQL query between broadcast, shuffle hash and sort merge? How can the spark UI or the query plan be interpreted?

smoortema · 10-22-2025

Let's say a table is configured to have automatic liquid clustering:ALTER TABLE table1 CLUSTER BY AUTO; How to know which columns were chosen by Databricks?

smoortema · 11-19-2025

Thanks for the useful informations! I have two additional questions:1. Your answer looks like it is LLM-generated. If it is, could you share which LLM you used for it?2. What is the best way to find the query in Spark UI? I am getting there through C...

smoortema · 10-30-2025

Thanks!What do you mean by this sentence? What do slots mean in this context?"we wire the sequential tasks in slots of 3 maybe which gives you a good balance but then you get 50 more of them and have to re-wire the whole situation again"

smoortema · 10-27-2025

Some questions related to your comment:1. In the original post, a blogpost was referenced where partition filtering was added not as a WHERE clause in a subquery (as you suggest), but in the ON clause of the MERGE INTO statement. Do you mean that the...

smoortema · 10-22-2025

From the documentation, it seems that in Python, there is such an option, only when creating or replacing a table.# To set clustering columns and auto, which serves as a way to give a hint # for the initial selection. df.writeTo(...).using("delta") ...

smoortema · 10-22-2025

Is there a way to set up automatic clustering by giving it a hint to use certain columns until Databricks has enough history to choose one?

Databricks Community

User Stats

User Activity

check statistics of clustering columns per file to see how liquid clustering works

Implement waiting logic in Databricks jobs

Why do I need MANAGE privilege on a connection to run a pipeline?

how to know which join type was used (broadcast, shuffle hash or sort merge join) for a query?

when automatic liquid clustering is enabled, how to know which columns are used for clustering?

Re: how to know which join type was used (broadcast, shuffle hash or sort merge join) for a query?

Re: Limiting concurrent tasks in a job

Re: How does partition pruning work on a merge into statement?

Re: when automatic liquid clustering is enabled, how to know which columns are used for clustering?

Re: when automatic liquid clustering is enabled, how to know which columns are used for clustering?