Databricks Community

thackman · 04-08-2025

Liquid clustering on integers or dates seems intuitive. But it's less clear how it would decide to partion files when the key column is a high cardinality string. Does it try the first character of the string and then if that's not unique enough it g...

thackman · 02-26-2025

Summary:We have a weird behavior with structs that we have been trying (unsuccessfully) to track down. We have a struct column in a silver table that should only have data for 1 in every 500 records. It's normally null. But for about 1 in every 50 r...

thackman · 07-10-2024

We have a job that runs on single user job compute because we've had compatibility issues switching to shared compute.Normally the cluster (1 driver,1 worker) takes five to six minutes to start. This is on Azure and we only include two small python l...

thackman · 07-10-2024

Our current system uses Databricks notebooks and we have some shared notebooks that define some python udfs. This was working for us until we tried to switch from single user clusters to shared clusters. Shared clusters and serverless now use Spark C...

thackman · 04-08-2025

We will need a bit more information. Are you asking: A) how an external user who is skilled at code can invoke a sql query via api. or B) how a non-technical external user can run a query via a simple ui?If it's option A: then you can create a person...

thackman · 04-08-2025

1) You might need to cache the dataframe so it's not recomputing for the write2) What type of cloud storage are you using? We've noticed slow delta writes as well. We are using Azure standard storage which is backed by spinning disks. It's limited to...

thackman · 07-15-2024

I'm not sure what you mean by "Ensure the Python binary's location is correctly set to resolve runtime issues" . We aren't using any binaries. Everything is just Databricks notebooks. In our case if we define a python udf function in the root notebo...

thackman · 07-15-2024

It's good to know that DBU charges don't start until the cluster is operational. But, we are paying Azure charges for the VMs and other infrastructure during that whole startup phase.

Databricks Community

User Stats

User Activity

How does liquid clustering handle high cardinality strings?

Inconsistant handling of null structs vs strucs with all null values.

Databricks cluster random slow start times.

Python udfs, Spark Connect, included modules. Compatibility issues with shared compute

Re: Allow an external user to query SQL table in Databricks

Re: Overwriting delta table takes lot of time

Re: Python udfs, Spark Connect, included modules. Compatibility issues with shared compute

Re: Databricks cluster random slow start times.