Data Engineering topics

Best practices for data quality in lakeflow

gowri_databrick — Tue, 01 Sep 2026 10:56:37 GMT

Hi everyone,

What are the recommended best practices for implementing data quality checks in Lake flow Spark Declarative Pipelines?

Should data quality expectations be applied mainly in the Bronze layer, Silver layer, or both?

Thanks!

Lakeflow connect

Rongali_Vasu — Tue, 01 Sep 2026 10:14:48 GMT

When can we expect lakeflow connect for (mysql,oracle,postgres) will be in GA from preview mode

How to calculate cost of each table for the specific Databricks Run ID

Dolly0503 — Tue, 01 Sep 2026 06:32:36 GMT

Hi Databricks Community,

I need help calculating table-level cost for each specific Job Run ID in Databricks.

I have multiple pipelines/jobs, and the same table can run multiple times with different Run IDs.

For example:
Job A
│
├── Run ID 1001
│ ├── Table A → $5
│ └── Table B → $3
│
├── Run ID 1002
│ ├── Table A → $7
│ └── Table C → $4
│
└── Run ID 1003
└── Table A → $6

I need the o/p to be

Job ID Run ID Table Table Cost

Job A	1001	Table A	$5
Job A	1001	Table B	$3
Job A	1002	Table A	$7
Job A	1002	Table C	$4
Job A	1003	Table A	$6

The key requirement is:

If a table was executed in a particular Run ID, I want to see the cost of that table specifically for that Run ID.

I do not want:

Table A = $18

without knowing which Run IDs contributed to that $18.

I want:

Table A
  Run 1001 → $5
  Run 1002 → $7
  Run 1003 → $6

I am currently exploring:

system.billing.usage
system.billing.list_prices
system.lakeflow.job_task_run_timeline
system.lakeflow.jobs

I would like to know the recommended Databricks approach to derive:

Job ID
→ Run ID
→ Task
→ Table
→ Table Cost

The solution should also ensure that the sum of table-level costs for a Run ID reconciles with the actual cost of that Run ID, without double counting.

What is the best Databricks-native way to achieve this table-level cost attribution by Run ID? and also i need end to end query

System generated Service Principal -App

rkhbo3003 — Tue, 01 Sep 2026 02:36:41 GMT

I created an App and creating agent to generate ETL pipeline through prompts. User interaction is with app, however when I do schema check on ADLS it says user does not have access. I cannot see SP in UI as it is system generated, even though I have provided all access to all workspace users, still issue exist.

Unable to create Compute

sanketpatel — Mon, 31 Aug 2026 20:03:14 GMT

Hello Community Support,

I am not able to create the compute resources with my Azure Databricks subscription. Can anyone help?

I was on a 14-day free premium trial plan. Now I am on Premium (+ Role-based access controls) plan. My azure account is pay as you go subscription plan.

My Workspace ID: 7405616306325796

OpenSharing ID: azure:eastus:6bad6940-d3cf-4e41-bf37-a18c6d615f22

Streaming tables fail with DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG after adding a

Oumeima — Mon, 31 Aug 2026 16:09:45 GMT

Environment:

Databricks Runtime: Current channel, Photon enabled
Edition: Pro
Platform: Azure

Description:

My setup consists of:

A managed ingestion pipeline for a SQL Server database that ingests raw data into bronze Delta tables (SCD Type 1), consisting of a gateway pipeline and an ingestion pipeline
A Spark Declarative Pipeline (SDP) that reads from those bronze tables via streaming and writes to silver tables

After adding a new column at the source, I performed a full refresh on both pipelines (otherwise the new column won't be ingested):

Full refresh of the managed ingestion pipeline (bronze) ✅
Normal refresh of the ingestion pipeline ✅ (necessary for data to actually be ingested in managed pipelines)
Full refresh of the SDP (silver) ✅

When a new normal pipeline update triggers, all silver streaming flows fail even though I performed the full refresh. The error is:

[DELTA_STREAMING_INCOMPATIBLE_SCHEMA_CHANGE_USE_SCHEMA_LOG] Streaming read is not supported on tables with read-incompatible schema changes (e.g. rename or drop or datatype changes). Please provide a 'schemaTrackingLocation' to enable non-additive schema evolution for Delta stream processing.

What I've tried:

The full refresh succeeds but every subsequent incremental/streaming run fails. The schema change seems to persist in the bronze tables' Delta transaction log even after a full refresh of the source (a hypothesis of mine)

Questions:

Why does the incompatible schema change persist in the Delta log after a full refresh of the source tables?
Is schemaTrackingLocation supported within SDP streaming table definitions, and how should it be configured? I believe it's automatically managed by the SDP pipeline and I shouldn't need to interfere with that.

Any guidance appreciated. Thanks!

Mismatch of spark.catalog.tableExists and SHOW TABLES in streams

i4ata — Mon, 31 Aug 2026 08:31:24 GMT

Consider this minimal example of a streaming workflow:

import os from pyspark.sql import functions as F, DataFrame from typing import Literal # Change these if you want to replicate CATALOG, SCHEMA = 'partner_demo_catalog', 'ad_sales_demo' spark.sql(f'use {CATALOG}.{SCHEMA}') log_file = os.path.join(os.getcwd(), 'debug.txt') def exists(table_name: str, method: Literal['spark', 'sql']) -> bool: # Bugged if method == 'spark': return spark.catalog.tableExists(table_name) # Working fine return spark.sql('show tables').filter(F.col('tableName') == table_name).count() > 0 def foreach_batch(batch_df: DataFrame, batch_id: int) -> None: # Write logs to a text file for simplicity with open(log_file, 'a') as f: f.write(f'batch_id: {batch_id}\n') # Check if the table is in the catalog if exists('t', 'sql'): f.write('Before stream, SQL: IN\n') else: f.write('Before stream, SQL: OUT\n') if exists('t', 'spark'): f.write('Before stream, Spark: IN!\n') else: f.write('Before stream, Spark: OUT!\n') # Write the table to the catalog batch_df.write.format("delta").mode('overwrite').saveAsTable('t') # Check if the table is in the catalog again if exists('t', 'sql'): f.write('After stream, SQL: IN\n') else: f.write('After stream, SQL: OUT\n') if exists('t', 'spark'): f.write('After stream, Spark: IN!\n') else: f.write('After stream, Spark: OUT!\n') f.write('\n') def test_stream(df: str, i: int) -> None: ( spark .readStream.table(df) .writeStream.foreachBatch(foreach_batch) # Pass a different checkpoint location just to trigger the stream again with no changes .option("checkpointLocation", f'{os.getcwd()}/checkpoints_test_{i}') .trigger(availableNow=True) .start() .awaitTermination() ) spark.sql('drop table if exists t') spark.sql('drop table if exists src') df_src=spark.createDataFrame([('Alice', 1)], ['name', 'age']) df_src.write.format("delta").mode('overwrite').saveAsTable('src') test_stream('src', 0) test_stream('src', 1)

As it can be seen, the script creates a toy table `src`, from which a stream is created and triggered twice. Inside `foreachBatch` I create a new toy table `t`. I check whether spark can see the new table in two ways: `spark.catalog.tableExists()` and Spark SQL's `SHOW TABLES`. For simplicity I am logging to a regular .txt file. The contents of it after running the script (using serverless compute and unity catalog) are the following:

batch_id: 0
Before stream, SQL: OUT
Before stream, Spark: OUT!
After stream, SQL: IN
After stream, Spark: IN!

batch_id: 0
Before stream, SQL: IN
Before stream, Spark: OUT! # Key line here
After stream, SQL: IN
After stream, Spark: IN!

This means that `spark.catalog.tableExists()` fails to see the table `t` even though it clearly exists in the catalog. Could anyone clarify what is going on and whether that behavior is intended? According to Genie, that is "a bug in the Spark Connect foreachBatch implementation".

I am using different checkpoint paths just so that I can trigger the full stream again without changing the source table. Still, using the same path and updating the table between streaming runs does not change the behavior. Using `batch_df`'s spark session has no effect either.

Best practice for dbutils with CICD

AdamIH123 — Sun, 30 Aug 2026 21:48:57 GMT

I have a job with the following base parameters. I use dbutils.widgets.get to access the parameter.

What is the best practice for developing with dbutils, especially when working locally in vs code or during CI? For example, IntelliSense and linting tools aren't aware of dbutils.

tasks:
- task_key: test_table
notebook_task:
notebook_path: ../src/test_table_dab.ipynb
base_parameters:
unity_catalog: ${var.unity_catalog}

Building a Delivery Assurance Agent: Predicting $8.4M in Penalty Risk with Databricks Genie

yasmeen — Fri, 28 Aug 2026 09:46:14 GMT

Delivery Assurance Agent: AI-Powered Risk Intelligence for Delivery Teams

The Problem
Delivery teams track thousands of tasks but struggle to answer:

Which client commitments will miss, and what should we do?

Traditional project tracking gives you task-level status but not commitment-level risk intelligence.

The Solution
We built the Delivery Assurance Agent - a Genie-powered platform that:
- Monitors 56 client commitments across 14 programmes
- Predicts slip probability using a 6-factor weighted risk model
- Quantifies $8.4M in expected penalty exposure
- Generates AI-drafted mitigation actions (Teams posts, Jira tickets, exec briefings)

Architecture

Data Layer: Unity Catalog (8 Tables)
- `commitment_risk` - One row per client commitment with slip probability
- `risk_factor_detail` - Six weighted factors per commitment (Throughput, Dependencies, Resources, Stakeholders, Cost, Quality)
- `milestone_forecast` - Phase gates with optimism gap detection
- `dependency_impact` - External blockers priced by commitment impact
- `leadership_actions` - Weekly decision queue
- `program_metrics` - Portfolio health (14 programmes)
- `velocity_trend` - Sprint-by-sprint throughput
- `action_queue` - Auditable log of generated artifacts

Intelligence Layer: Genie Space
- 13 benchmark queries
- 8 curated starter questions for executives
- Entity matching on 28 columns
- 60+ documented columns with business context
- SQL examples teaching risk model patterns

Application Layer: Databricks Apps V2
- FastAPI backend with service principal execution
- Genie API integration for conversational queries
- SQL Warehouse for KPI dashboard
- Model Serving (Llama 4) for action generation
- OAuth2 for user context

The Risk Model

Six Weighted Factors (Logistic Regression):
1. Throughput(1.75) - Current velocity vs. remaining work
2. Dependencies(1.20) - External blockers and late deliverables
3. Stakeholder Signal (0.80) - Client escalations, team concerns
4. Resources(0.75) - Staffing gaps, attrition
5. Cost Performance(0.70) - Budget overruns (CPI < 1.0)
6. Quality (0.62) - Defect rates, rework cycles

Intercept: -3.2

Formula: `slip_probability = logistic(Σ(factor × weight) - 3.2)`

Accuracy: 92% on 25 delivered commitments (50% threshold, 5-day tolerance)

Key Features

1. Executive Intelligence

Ask: "What is the total expected penalty exposure?"
- 31 commitments at risk
- $8.4M expected exposure (probability-weighted)
- $18.9M worst-case (if all miss)
- Average 52% slip probability

2. Root Cause Analysis
Ask: "Why is the Atlas Core Banking Migration at risk?"
- Throughput: 47% contribution (velocity can't close remaining work)
- Dependencies: 31% contribution (three teams late)
- Named drivers, quantified shares, responsible owners

3. Optimism Gap Detection

Ask: "Where are reported dates not supported by data?"
- Compares forecast_date (what PM reports) vs. predicted_date (what data implies)
- Surfaces commitments where optimism gap > 7 days

4. AI Action Generator
The app generates data-grounded artifacts:
- Teams notifications: Bold headline, 3-4 evidence bullets, clear ask
- Jira tickets: JSON payload with priority, assignee, acceptance criteria
- Executive summaries: Board-ready briefing (under 200 words)
- Mitigation plans: Sequenced actions with owners and dates

All artifacts logged to `action_queue` with the model endpoint that produced them.

Databricks Features Used
1. AI/BI Genie - Natural language to SQL
2. Unity Catalog - Governed 8-table data model
3. SQL Warehouses - Serverless query execution
4. Databricks Apps V2 - Production app hosting
5. Databricks SDK - Programmatic Genie + SQL integration
6. Model Serving- AI action generation (Llama 4)
7. Service Principals - Consistent app-level permissions

Results
- ✅ 100% benchmark pass (13/13 queries)
- ✅ 60+ columns documented with entity matching
- ✅ 8 starter questions for immediate value
- ✅ Responsible AI: "AI-generated risk assessment. Verify before action."
- ✅ Production-ready: Service principal execution, auditable queue

Key Takeaways
1. Genie as a Platform: Not just Q&A - integrate with KPIs, actions, and workflows
2. Model Interpretability: Six named factors beat a black box
3. Probability-Weighted Exposure: Rank by expected_penalty × slip_probability, not worst-case
4. Responsible AI: Always include verification guidance for high-stakes decisions

Try It Yourself
The pattern is reusable for any domain with:
- Commitments/deadlines you must hit
- Multiple risk factors you can measure
- Financial or reputational consequences of missing

Links
- Live App: [View Demo](https://delivery-assurance-agent-7474648612775687.aws.databricksapps.com)

Making transform dynamic and user-selectable

treesloth — Thu, 27 Aug 2026 21:35:01 GMT

Hello. I have a straightforward bar graph visual. I generally want to display one year's data, displayed monthly. However, when certain filters are applied, the data becomes limited to 1 month, which just produced one very wide bar. Not very useful. So, I'd like the option for a user to switch that to a daily view instead. It's simple enough to present a filter to the user containing "Monthly" and "Daily", but is there a way to make the visual use that as its transform? Thank you.

Suggestions for recommendation

ThiamLee — Thu, 27 Aug 2026 10:08:33 GMT

Hey everyone! 👋 I’m currently looking for some good tools for data analysis. If you have any recommendations, please share them with me. Would really appreciate it! 😊

Title: Oracle CDC pipeline (Lakeflow Connect) never terminates

Ashoka — Wed, 26 Aug 2026 04:59:20 GMT

Setup: Oracle 21c XE (CDB/PDB), Direct CDC Extraction, no gateway, triggered pipeline on hourly schedule, single table.

Problem: A triggered run has been active 95+ minutes with no work left. All flows report IDLE, waiting for new data, the source is idle, and target data is already fully correct. But the run won't stop.

Event log:

Code

Direct CDC Extraction is IN_PROGRESS, running for 92 minutes.
Zero snapshots are pending. Will stop once the lag converges.
Current CDC Ingestion Lag : 19949 seconds.
Lag is increasing, perfectly linearly:

Running (min) Lag (sec)
40 16829
63 18149
75 18869
92 19949
Exactly 360 sec of lag per 360 sec elapsed — 1:1 with wall clock. Reported lag (~5.5 h) also far exceeds the run duration (92 min), which suggests it's computed as now() − timestamp of last processed change rather than against the current redo position. If so, the exit condition can never be satisfied while the source is quiet.

Run sequence (this part seems diagnostic):

Run 1 — full refresh, 200K rows → completed
Run 2 — incremental, zero source changes → completed
Run 3 — incremental after 10K inserts + 100K deletes (batched, ~3.25 GB redo) → hangs
So the idle run finished fine; the busy one didn't.

Source side is clean: ARCHIVELOG + PK supplemental logging on, 50 archived logs on disk (nothing purged), FRA at 33%, listener up. All 20 replication-user sessions INACTIVE on SQL*Net message from client — Oracle is waiting on the client, nothing executing. Those 20 sessions also never get released between runs.

Impact: hourly schedule is dead — the next run skipped with Max Concurrent Runs, and every subsequent one will too.

Questions:

How is Cdc Ingestion Lag calculated? Is it expected to grow while the source is idle?
What's the actual convergence threshold that ends a triggered run?
Any idle-detection or timeout for triggered runs with no pending changes?
Is a periodic heartbeat write a known workaround, or is there a setting?
Are replication-user sessions meant to be pooled/reused across runs?
Anyone seen this?

databricks SQL UDF in select statement

pepco — Tue, 25 Aug 2026 08:34:01 GMT

In the Unity Catalog we can now create/register SQL UDFs. There are two types - one that returns table and other that returns just a value. If the function that returns value is based on the SQL query and joins it would in standard relational databases represent a correlated query executed for each row - which is usually a code smell in the relational databases.

A very simple example (I just made it up):

select b.value from table1 a join table2 b on 1 = 1 and b.col = a.id and b.col = p_parameter1 join table 3 c on 1 = 1 and c.col = p_parameter2 and ...

The function that returns single value based on simple logic; i.e. based on the input parameters is during execution extrapolated into the query. Basically, it's super helpful for hiding complex case statements or amounts recalculation using the same logic.

Since Databricks is using columnar storage, how does it behave for functions that contain joins? Does it also expand the underlying query into the main query? If yes, does it mean that for every row it adds underlying query?

Bug in Databricks Assets/Automate Bundles

EhuamanS — Mon, 24 Aug 2026 23:21:52 GMT

Hi everyone so it happens that I was working normally in the UI, adding jobs, deploying and so and then suddenly I could not add any existing job to my bundle because the dropdown doesn't show any job anymore and I have a bunch of jobs that hasn't been added to the bundle yet. I think that is bug because before it was working well. I would be really glad if someone could help me and explain this behaviour to me, I don't know what happened.

Declarative Automation Bundle - How to handle Managed resources?

ChristianRRL — Mon, 24 Aug 2026 20:39:05 GMT

Hi there,

I'm making a new repo from scratch to make it as DAB-compatible as possible. I was considering using DAB for the creation/management of resources such as schemas and volumes.. however, a concern crossed my mind. If these resources are MANAGED, then if someone were to run the command:

databricks bundle destroy

This would be a huge risk if this automatically deletes all of the underlying data.

I'm curious to ask, based on this community what is the suggested practice for using (or not using) DABs for schemas/volumes? Below is a possible example of our volume data being MANAGED:

Distinguishing Runs Related to an All-Purpose Cluster

DB1To3 — Mon, 24 Aug 2026 19:55:05 GMT

Sorry for the basic question. If I am using an all-purpose clusters in Databricks, it seems to combine all my runs together and execute them within the same Apache Spark application. There is only one application shown in the Spark UI. Unrelated runs gets mixed up together in the same Spark UI. I'm using an all-purpose cluster, not a job cluster.

Normally I send the job runs to this cluster by way of the REST API (jobs api).

How can distinguish and review the list of runs that are currently associated with a given "all-purpose" cluster? I'd like to be able to at least get a simple list of active runs on a given compute. There are different runs using different compute, and I haven't found a way to filter the list of runs, based on the related compute cluster. What am I missing?

Lakehouse Federation (Snowflake) — large query results fail to download from internal stage

margarita_shir — Mon, 24 Aug 2026 19:01:39 GMT

We're hitting a consistent failure with Lakehouse Federation to Snowflake where large result sets fail during the result-chunk download, while small results work. Looking for help isolating whether this is the Databricks Snowflake connector or a config issue on our side.

Setup
- Databricks on AWS, customer-managed VPC, us-east-1.
- Snowflake foreign catalog; account reached over PrivateLink (…privatelink.snowflakecomputing.com).
- Reproduced on a SQL warehouse, both Current and Preview channels.

Behavior (size-dependent):
- SELECT * FROM <table> LIMIT 5 → returns fine.
- SELECT * FROM <table> LIMIT 100 → hangs, then errors.

Error:
JDBC driver internal error: Max retry reached for the download of chunk#0
(Total chunks: 1) retry: 7, error: net.snowflake.client.jdbc.SnowflakeSQLException:
JDBC driver encountered communication error. Message: Exception encountered for
HTTP request: sfc-va2-ds1-18-customer-stage.s3.amazonaws.com:443 failed to respond.
at net.snowflake.client.jdbc.RestRequest.executeWithRetries(RestRequest.java:1082)

What we've established:
- The query succeeds on Snowflake's side (confirmed via their query history — it scans and returns rows); the client then cancels after it can't pull the result chunk. So this is the client-side download of result chunks from Snowflake's internal S3 stage, not permissions or query logic.
- Small results come back inline and work; large results stage to S3 and the connector fails on chunk#0. That isolates it to the internal-stage result-chunk download path.
- Network is not blocking it: from the compute, a direct request to the stage bucket (sfc-va2-ds1-18-customer-stage.s3.amazonaws.com) returns a clean HTTP 403 (reached S3, S3 responded). We have an S3 gateway endpoint with a wide-open policy, and the subnets route the us-east-1 S3 prefix list to it.
- Not a driver-version issue: it reproduces on the Preview channel (newer connector) as well as Current.

Is this a known issue with the Snowflake federation connector on a PrivateLink account connection + public-S3 internal stages, and is there a recommended fix or workaround?

INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted

emorgoch — Mon, 24 Aug 2026 14:50:55 GMT

I've got an ADLS Gen 2 container that has cloud files that I want to ingest with AutoLoader. The files are saved into the root of the container.

I created an external location for that container to be able to access it through managed credentials. However, in my script used to set it up, I also accidentally created a catalog, schema, and table as a test to validate access. No problem, I thought. I deleted the catalog within databricks, and then deleted __unitystorage directory & files.

However, now when I try to have autoloader access the files in that container I'm getting the following error:

AnalysisException: [RequestId=8f219926-d946-4601-a683-18495a6ded8f ErrorClass=INVALID_PARAMETER_VALUE.LOCATION_OVERLAP] Input path url 'abfss://container@mystorageaccount.dfs.core.windows.net/' overlaps with managed storage within 'CheckPathAccess' call.

If I try and delete the external location from Unity Catalog, Databricks prompts me that it can't delete it due to the location having 1 dependent managed tables.

I used the Force delete option, and then re-added the external location, and I'm still getting LOCATION_OVERLAP exception when trying to read the files with AutoLoader.

I tried using SHOW DROPPED TABLES, but that doesn't work cause as the catalog was dropped first.

Does anyone have any suggestions on how I can get UC to recognize that this container root is no longer being used for any managed resources?

How to show full query results in a SQL alert's notification?

SRJDB — Mon, 24 Aug 2026 14:25:34 GMT

I have a SQL alert; the code behind it looks like this:

SELECT COLUMN_1, COLUMN_2, COUNT(*) AS TOTAL, 1 AS FORCE_TRIGGER
FROM MY_TABLE
GROUP BY COLUMN_1, COLUMN_2

The idea is that a scheduled job updates my_table and then runs the alert. The alert triggers when MIN(FORCE_TRIGGER) > 0 (in other words, each time it runs). It then sends me a notification via email with the query results included via the variable @QUERY_RESULT_TABLE.

The alert runs successfully and sends me an email. However, in it I only get a single row of query results.

Does anyone know how to get the full query results to display in the email please?

cross-region DR in Azure Databricks (24h RPO/RTO)

Deny1 — Mon, 24 Aug 2026 08:53:47 GMT

Hi,

We're designing a DR strategy for an Azure Databricks platform and would appreciate guidance on current best practices for achieving approximately 24-hour RPO and RTO across Azure regions.

Our platform includes Unity Catalog, DAB, Jobs, Notebooks, SQL Warehouses, Lakeflow Declarative Pipelines, Materialized Views, Streaming Tables, Delta Sharing, and related security/governance components.

Our current approach is:

Primary workspace in North Europe
Standby workspace in West Europe
Separate metastores per region
DAB deployment to both workspaces
DR resources kept paused until failover
Nightly replication of business-critical data
Recovery through a combination of replication, replay, and recomputation

A few questions:

Is a warm-standby workspace the recommended pattern today?
What is the preferred approach for Unity Catalog and metadata across regions?
How are customers handling Lakeflow Pipelines, Streaming Tables, and Materialized Views given the lack of native cross-region replication?
Is replay/recomputation the recommended recovery model for pipeline outputs?
What are common approaches for SQL Warehouses, dashboards, alerts, and region-specific IDs?
Are there reference architectures or proven customer patterns for 24h RPO/RTO on Azure Databricks?
What DR limitations should we account for regarding Managed Volumes, MLflow models, Delta Shares, Vector Search, Genie assets, and similar services?

We're interested in both official Databricks guidance and real-world implementations from other customers.

Thanks in advance.