- 204 Views
- 4 replies
- 0 kudos
Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type
Data Ingestion (Lakeflow) Feature Bug:While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default da...
- 204 Views
- 4 replies
- 0 kudos
- 86 Views
- 2 replies
- 1 kudos
Unable to Attach Databricks Serverless Compute – Environment Version 5 (Heavy Load Warning)
Hi Databricks Community,I'm experiencing an issue with Databricks Serverless Compute while trying to attach it to my notebook using Serverless Environment Version 5.Whenever I attempt to attach the compute, I receive the following warning: "Databrick...
- 86 Views
- 2 replies
- 1 kudos
- 1 kudos
Hello @NaniOnyx07, I took a look at both internal and external documentation and here is what I found. First, credit to @ThomazNeto for the thorough first pass. I checked his facts against the docs and he is spot on: environment version 5 shipped in ...
- 1 kudos
- 386 Views
- 4 replies
- 3 kudos
Unity Catalog external location access error after metastore attachment
Hi everyone,I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.Here are the symptoms:The Setup: Storage credentials and an external location were successfully created, and...
- 386 Views
- 4 replies
- 3 kudos
- 3 kudos
Hello @Khasim_1, I took a look at both internal and external documentation and here is what I found. First, credit to @data_pulse, @pradeep_singh and @saurabh18cs, who've already laid out the standard triage. Capture the exact error. Confirm the job'...
- 3 kudos
- 289 Views
- 7 replies
- 9 kudos
How to Connect Databricks to Snowflake & Query Live Data?
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
- 289 Views
- 7 replies
- 9 kudos
- 9 kudos
Hello @Raj_DB, I took a look at both internal and external documentation and here is what I found. The thread already has the right answer: Lakehouse Federation. @saurabh18cs, @Harshad_B, @AbhilashNagilla, @Aravind_Reddy, @Jen_Evangelista and @balaji...
- 9 kudos
- 96 Views
- 3 replies
- 0 kudos
Resolved! Schema evolution: new NOT NULL column fails on existing Silver data (Lakeflow Connect source)
Hi everyone,We ingest data from our source database with Lakeflow Connect, through a foreign catalog, into a Bronze layer. In the Silver layer, we apply the correct data types and constraints based on our data dictionary.Recently, some tables in the ...
- 96 Views
- 3 replies
- 0 kudos
- 0 kudos
Good points already shared on the nullable → backfill → NOT NULL approach. I'd add a few Lakeflow Connect and Silver recovery considerations, particularly to avoid unnecessary full refreshes.1. Lakeflow Connect: Verify historical data before backfill...
- 0 kudos
- 188 Views
- 5 replies
- 6 kudos
Resolved! How do you collect Excel files from business users into Databricks ?
Hi everyone,I build reports in Databricks from Excel files maintained by many different business users. The files are varied and come from different teams, and I haven’t settled on an approach yet.One idea I’m considering: create a landing location w...
- 188 Views
- 5 replies
- 6 kudos
- 6 kudos
You can use a Unity Catalog Volume backed by S3 as the landing zone and its the right way. You can give each source team their own folder in the volume, allowing users to drop files through the UI. Pipeline can pick up files on arrival using a job fi...
- 6 kudos
- 173 Views
- 2 replies
- 1 kudos
guidance on cleainig up after long cdc failure
Hi All,We had a CDC failure which took far to long to resolve. A conversation with vendors on that.Does anyone have any guidance on the process steps to identifying the missed changes and applying them to the tables.I have so far run a query to ident...
- 173 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, @turagittech ,Your source-to-target comparison is a good starting point. Check missing inserts, changed values, and records deleted from the source separately. A source-to-target left join alone can miss deletes.Since you mentioned a full refresh...
- 1 kudos
- 245 Views
- 3 replies
- 0 kudos
Zerobus support for private-endpoint storage (MLflow traces to Unity Catalog)
We'd like to store MLflow 3 traces in Unity Catalog, using an experiment linked to a UC table-prefix trace location. In our Azure UK South workspace, linking the location fails with:PERMISSION_DENIED: Zerobus failed to open the table. Make sure the t...
- 245 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @jAAmes_bentley Following up on the two answers above with the doc lines behind them, one Azure change that affects your third question, and an option for now. Thank you for quoting the full error and stating the storage policy up front. That ...
- 0 kudos
- 1284 Views
- 18 replies
- 4 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 1284 Views
- 18 replies
- 4 kudos
- 4 kudos
Hi @dbernstein_tp,Good to hear the MERGE is only touching a few thousand rows now. That's the part that matters for Direct Lake. On the job vs manual timing, the same code running at very different speeds almost always means it isn't running on the s...
- 4 kudos
- 177 Views
- 3 replies
- 2 kudos
Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation
I'm using Databricks Free Edition and trying to configure an AWS S3 external location through Unity Catalog.My storage credential validation consistently fails at Assume Role, while Self Assume Role and External ID Condition are skipped.Error: Failed...
- 177 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ifra_inam ,The PERMISSION_DENIED failure at the very first Assume Role stage means AWS IAM is instantly rejecting the cross-account trust handshake. Because the failure happens immediately, Databricks never even reaches the S3 permissions check.C...
- 2 kudos
- 61 Views
- 0 replies
- 0 kudos
From the Databricks Hackathon to the Databricks Community!
Back in March 2026, I participated in a Databricks hackathon, where I had the opportunity to build a solution using Databricks and explore its capabilities in solving real-world problems.I'm excited to share that my project secured a place among the ...
- 61 Views
- 0 replies
- 0 kudos
- 265 Views
- 3 replies
- 2 kudos
Resolved! Update Ingestion pipleline to use pager duty with API
Hi,I have been working to update a cdc ingestion pipeline to add a notification to it on failure. We have had two instances where a maintenance task by vendor as tripped up CDC with some grief.I reviewed all the information and I am getting an error ...
- 265 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @turagittech, a pipeline's [notifications](https://docs.databricks.com/api/pipelines/v2/edit) setting accepts only email_recipients and alerts, while notification destinations such as PagerDuty are added in a job's notification settings. For the ...
- 2 kudos
- 118 Views
- 0 replies
- 1 kudos
Lakeflow Expectations: Dropping Bad Rows Can Hide Real Sales
A practical data quality lesson: choose the response to a failed rule based on its business impact.Consider a hypothetical retailer using Azure Databricks. An incoming order has a valid order ID and sales amount, but its customer ID is missing.A rul...
- 118 Views
- 0 replies
- 1 kudos
- 267 Views
- 3 replies
- 0 kudos
Real time Streaming to AI Engine at Databricks
Hi there,Usercase: Fraud Detection Technology : Databricks, Kafka, Microsoft Foundry, Dashboard showing real-time reportsAsk: Please let me know if any reference or technology details to achieve 1-2 milli second (Max) latency using above technology s...
- 267 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @abhietc31, the fraud detection Solution Accelerator scores Kafka transactions in real-time mode and feeds a live Databricks Apps dashboard from Lakebase. For your own pipeline, the Lakebase sink writes real-time mode output to a Lakebase table wi...
- 0 kudos
- 72 Views
- 0 replies
- 0 kudos
Structured Streaming: Choose Watermarks Around Real Arrival Delays
Structured Streaming: Choose Watermarks Around Real Arrival DelaysA practical streaming lesson: measure how late business events arrive before choosing a watermark threshold.Consider a hypothetical retailer using Azure Databricks to calculate sales i...
- 72 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
