Databricks Community

Hi @arushigulati, Lakebridge (the Databricks Labs project formerly known as Remorph) does support Oracle as a source dialect for transpilation, but the DDL handling, particularly around constraints like PRIMARY KEY, has some gaps depending on the ver...

Hi @echol, This is a common scenario when multiple team members work with Databricks Asset Bundles, and there are a few approaches to solve it cleanly. THE ROOT CAUSE When Staff A deploys a bundle, the jobs and other resources are created with Staff ...

Hi @RIDBX, Databricks Lakeflow Jobs has several features that let you build exactly this kind of tiered, dependency-driven orchestration natively. Here is how I would approach your HR (Tier 1) and Finance (Tier 2) scenario. OPTION 1: SINGLE ORCHESTRA...

Hi @_its_akshaye, The pipeline event log captures exactly what you need. Every Lakeflow Spark Declarative Pipeline (SDP, formerly known as DLT) records flow_progress events that include per-flow metrics with num_upserted_rows and num_deleted_rows fie...

Hi @aranjan99, There are two separate topics here, so let me address each one. WHY THE QUERY IS SLOW (~50 SECONDS ON X-SMALL) System tables are served via Delta Sharing from a Databricks-hosted storage account in the same region as your Unity Catalog...

Hi @dpc, Unity Catalog does support group ownership of objects, including groups synced from Azure AD (Entra ID) or other identity providers. This is fully available today and addresses each of the scenarios you described. TRANSFERRING OWNERSHIP TO A...

Hi @fintech_latency, This is a common architectural challenge when you need sub-second or near-real-time event processing. Let me walk through your three questions and then outline the recommended patterns. QUESTION 1: CAN YOU GUARANTEE FIXED WARM SE...

Hi @sandy_123, The "Multiple failures in stage materialization" error occurs when Spark executors fail to fetch shuffle data from other executors during a shuffle stage. With 30 million rows, your job is likely hitting resource limits during a shuffl...

Hi @ajay_wavicle, Migrating UC managed tables between workspaces while preserving delta history requires a different approach than external tables, because Unity Catalog controls the underlying storage for managed tables and you do not have direct ac...

Hi @Dhruv-22, This is a valid use case and your workaround is solid. Let me share some context on the current behavior and a few approaches that may help streamline things. CURRENT BEHAVIOR WITH SCHEMA EVOLUTION When automatic schema evolution is ena...

Hi @SatabrataMuduli, The ORA-12170 TCP connect timeout error is a networking issue, not a driver or credentials problem. Your Databricks cluster cannot reach the Oracle host and port within the 15-second TCP timeout window. The fact that it works fro...

Hi @Digvijay_11, Here are answers to each of your three questions about Lakeflow Spark Declarative Pipelines (SDP): 1. RUNNING SDP PIPELINES IN PARALLEL WITH DYNAMIC PARAMETERS SDP automatically determines the dependency graph across your table and v...

Hi @GergoBo, There are a few approaches depending on whether you need playback in a notebook or within a Dash/Flask web app. Here is a breakdown of each scenario. OPTION 1: VIDEO PLAYBACK IN A DATABRICKS NOTEBOOK For notebooks, you can read the MP4 f...