Databricks Community

I would implement stateful streaming by using transformWithStateInPandas to keep the state and implement the logic there. I would avoid doing stream-stream JOINs.

It should be still supported, the only change I can find is that the table that you're referencing can't have a row filter applied to it. Has the users table got any type of row filter on it? It also can't create any type of circular reference.

Hi, There are a few things that could cause the issues you've outlined: Temporary resource scarcity (compute, network bandwidth, or concurrency limits on the warehouse) can lead to queries taking longer than usual or connections being dropped u...

Yes, this is a common challenge. Most teams avoid 1:1 conversion and instead rationalize ETLs first, then rebuild using Spark-native patterns (joins, lookups, SCDs) with strong validation. We’ve seen good results combining this approach with Kanerika...

Hi Chirag. At Kanerika Inc,, we've built a migration accelerator that automates 80% of the Informatica to Databricks migration process, saving you significant time, effort, and resources. You can check out the demo video of the same here - https://ww...

This is a common challenge in enterprise SAP Datasphere and Databricks integrations, particularly in brownfield, cross-cloud setups. We’ve seen multiple cases where sharing between SAP and Databricks works as expected, while the reverse path introduc...

From community experience, vector index sync behavior depends heavily on how the Delta table is updated. With OVERWRITE, the table is effectively replaced, so the vector index typically treats this as a full refresh. Existing embeddings are dropped a...

This is a well-documented issue, and it highlights how dependency conflicts in managed runtimes can quickly become a blocker for teams extending Spark beyond standard use cases.We see similar challenges when organizations build custom providers or in...

I have been using and implementing UC in various workspaces across industry, BYOL is the one I am really looking forward to implement next.Thanks @AbhaySingh for consolidating it here.

@bercaakbayir - 2 areas to look at for permissions.Unity Catalog PermissionDestination‑level permissionsPlease Check,UC enabled for your workspace. [Metastore Admin, not workspace Admin]CREATE permissions on the target catalog - User or SP should hav...

found a working example -databricks pipelines update <pipeline_id> --json @new_config.jsondatabricks pipelines start-update <Pipelineid>where in use JSON for passing parameters.. every run update the parameters with new json file

Hello @jlancaster86! Thanks for flagging this. Could you please raise a ticket with the Databricks Support team mentioning the missing link in the course, so they can get back to you directly with an update? I’ve also shared this internally.

@libpekin - Short Answer - its AWS-only and no such 'automated' path/choice to migrate to Azure.