Databricks Community

Big question, but in my experience it comes down to five decisions. Opinionated take:1. Put a buffer in front, always. Kafka/Kinesis/Event Hubs between producers and Spark. The broker absorbs the spike; your pipeline drains at its own pace. A pipelin...

Good points, Brahmareddy — and I think we mostly agree, with one nuance worth teasing out on the cross-team part.A published streaming table already *is* a stable Delta table from the reader's perspective — consumers query it by its UC name and never...

Can u try this in your toml file? [tool.databricks.flint] ignore = ["SCPAP005"]

Answers: Automated curation trade-off: the upside is low setup; the downside is less explicit control. Genie ranks and conflict-resolves snippets automatically, so if your sources are inconsistent or ambiguous, you still need to review and clean up ...

Hi,   Just been looking into this for you. All the docs I found suggested it would reject to I tried testing it myself. I tried setting traffic_config with various percentage combinations that don't sum to 100, on both Azure and AWS: Config (A / B) ...

Honest answers first, then a debugging path — because part of this simply isn't documented, and I'd rather say so than guess.1. Is the SSM Agent expected in DBR images? I found no Databricks documentation committing to include it, and your own findin...

The pattern is documented and you are right not to full refresh pipeline 2, but as written the deletion detection can never fire, and the full refresh is not why. spark.readStream.table("raw.contact") makes contact append-only, so no key ever leaves ...

Nice progress — with compute ACTIVE and the app RUNNING, we've eliminated the original cause and narrowed this down to the auth/access layer. A few thoughts:First, one thing in your test that's easy to over-read: the Python requests behavior is actua...

Hi @ChristianRRL  I tried with FHIR and it worked. The process is the same for every source. To do GitHub instead of the FHIR example, you change exactly three things, the connector name (github), the credential (a GitHub token), and the list of tabl...

Hi skipowder99,When you set a DEFAULT COLLATION at the catalog or schema level, it is forward looking. It does not retroactively apply to existing tables, columns or views. It leaves you with a split environment the newly created objects default to U...