Databricks Community

@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.(1) Driver logs a...

Good question, and the answer is usually simpler than it looks: those secrets are almost always YOUR OWN credentials for whatever third-party service the app talks to — not something the developer sends you.Here's the model: Marketplace apps aren't a...

I ran some tests.**Answer — verified at source level and across six runs, Spark 3.5.2 → 4.2.0(evidence dossier attached):**There are two independent rejections here sharing one error class, which iswhat made this confusing:1. `CAST(struct AS VARIANT)...

Hi @yutmurakami — I can now answer your question with data instead of hypotheses. I ran your exact scenario on my own Free Edition workspace (also AWS, different workspace than yours), twice.Test 1: existing app on my Free Edition workspace — Running...

We hit exactly this, offline feature store on Redshift, ~200-column denormalized table fed by domain models. Adding a feature meant NULLs for history, and full recompute was too expensive, so we landed on: recompute the domain for the backfill range ...

Great topic! Databricks machine creation and setup can seem complex at first, but once you understand clusters, compute resources, and workspace configuration, it becomes much easier. A step-by-step guide with screenshots and best practices for begin...

If your action (count(), collect(), show(), etc.) is already inside the try block, this sounds like a false positive. The warning is likely from a Databricks inspection rather than the Flint time-series library. Check if inspection SCPAP005 can be di...

Good points, Brahmareddy — and I think we mostly agree, with one nuance worth teasing out on the cross-team part.A published streaming table already *is* a stable Delta table from the reader's perspective — consumers query it by its UC name and never...

Hi @ChristianRRL  I tried with FHIR and it worked. The process is the same for every source. To do GitHub instead of the FHIR example, you change exactly three things, the connector name (github), the credential (a GitHub token), and the list of tabl...

Hi skipowder99,When you set a DEFAULT COLLATION at the catalog or schema level, it is forward looking. It does not retroactively apply to existing tables, columns or views. It leaves you with a split environment the newly created objects default to U...