Databricks Community

Hi @IM_01, The public guidance points to one natural-language question or title per example SQL query, rather than multiple sample questions attached to a single query. In the Tune Genie Space quality docs, Databricks says that for each example SQL q...

Although the PySpark documentation states that DataFrame.foreachPartition() is a shorthand for df.rdd. foreachPartition(), there is an important difference when running on Databricks shared clusters (especially with Unity Catalog and Spark Connect).D...

This issue appears to be related to Azure Storage access through Unity Catalog rather than the data itself, especially since the same workload was working fine with Hive and the failure is intermittent.A few areas worth checking:1. Storage Credential...

Hi , While DESCRIBE DETAIL gives you the total sizeInBytes, it lumps everything together. Neither ANALYZE nor the Information Schema will break down the exact physical storage overhead of your Change Data Feed (CDF) or your Delta logs.Because a Delta...

Great write up and felt useful. Thanks for sharing the real experience.!

Yes, you can achieve this seamlessly, but not by overriding the custom_tags inside the cluster variable. Instead, you define your specific tags at the Job level, and Databricks automatically merges them with your cluster variable's tags.Because compl...

Hi @bi_123, I would avoid parsing the full rendered UNKNOWN_FIELD_EXCEPTION message. Databricks explicitly notes in the error-handling documentation that the rendered and parameterised messages are not stable across releases, so any logic that depend...

Hi @damodhargandha, A good way to think about a shallow clone is that it copies the table’s metadata but still points to the source table's data files rather than copying them. Databricks explains that behaviour in the Clone a table " section of the ...