- 247 Views
- 3 replies
- 1 kudos
guidance on cleainig up after long cdc failure
Hi All,We had a CDC failure which took far to long to resolve. A conversation with vendors on that.Does anyone have any guidance on the process steps to identifying the missed changes and applying them to the tables.I have so far run a query to ident...
- 247 Views
- 3 replies
- 1 kudos
- 1 kudos
Checked the full refresh.I can easily control pauses as we only update end user tables one a dayIt is SQL Server CDC. Full refresh completed Ok
- 1 kudos
- 475 Views
- 5 replies
- 3 kudos
Unity Catalog external location access error after metastore attachment
Hi everyone,I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.Here are the symptoms:The Setup: Storage credentials and an external location were successfully created, and...
- 475 Views
- 5 replies
- 3 kudos
- 3 kudos
This type of issue can sometimes occur when AWS IAM permissions are correctly configured, but Unity Catalog is unable to authorize access through the storage credential or external location. I would recommend checking a few things before changing the...
- 3 kudos
- 771 Views
- 10 replies
- 5 kudos
Resolved! Materialized Views - Strange Difference in Refresh Performance
I have built the same materialized view twice with only one difference: mv01 does incremental refresh and mv02 does full refresh.The mv reads from 7 tables and 1 mv, performs unions, joins, anti-joins as well as filters, and calculates columns on the...
- 771 Views
- 10 replies
- 5 kudos
- 5 kudos
The incremental refresh may actually be doing more work because of the complexity of the query and how changes propagate through joins, unions, and anti-joins. I’d be interested to know whether the query plan shows expensive change detection or recom...
- 5 kudos
- 618 Views
- 7 replies
- 3 kudos
Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves
From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the ...
- 618 Views
- 7 replies
- 3 kudos
- 3 kudos
Hello @sudopradip, I took a look at both internal and external documentation and here is what I found. Credit first. @SumeshKashyap had the practical answer early: write through /Volumes or the Files API from inside Databricks, don’t pin the public I...
- 3 kudos
- 3266 Views
- 5 replies
- 2 kudos
Time Travel Error when selecting from materialized view (Azure Databricks)
Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...
- 3266 Views
- 5 replies
- 2 kudos
- 2 kudos
This is an interesting discussion about handling time travel errors when working with materialized views in Azure Databricks. Understanding how different data objects support historical queries is important, especially when troubleshooting complex da...
- 2 kudos
- 88 Views
- 0 replies
- 0 kudos
ASF Corporation
Hello Databricks Community,I'm representing ASF CORPORATION, a piling, foundation engineering and construction company established in 2017 and based in Dhaka, Bangladesh.Our company specializes in hydraulic rotary piling, rotary drilling, auger borin...
- 88 Views
- 0 replies
- 0 kudos
- 1450 Views
- 19 replies
- 7 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 1450 Views
- 19 replies
- 7 kudos
- 7 kudos
Hi @dbernstein_tp,Thanks for testing all of that and reporting back. Good to know deletion vectors didn't upset Direct Lake.I think your two findings are really the same problem. If most of the gold MVs are fully recomputed on every update, each refr...
- 7 kudos
- 329 Views
- 4 replies
- 0 kudos
Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type
Data Ingestion (Lakeflow) Feature Bug:While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default da...
- 329 Views
- 4 replies
- 0 kudos
- 168 Views
- 2 replies
- 2 kudos
Unable to Attach Databricks Serverless Compute – Environment Version 5 (Heavy Load Warning)
Hi Databricks Community,I'm experiencing an issue with Databricks Serverless Compute while trying to attach it to my notebook using Serverless Environment Version 5.Whenever I attempt to attach the compute, I receive the following warning: "Databrick...
- 168 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @NaniOnyx07, I took a look at both internal and external documentation and here is what I found. First, credit to @ThomazNeto for the thorough first pass. I checked his facts against the docs and he is spot on: environment version 5 shipped in ...
- 2 kudos
- 428 Views
- 7 replies
- 9 kudos
How to Connect Databricks to Snowflake & Query Live Data?
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
- 428 Views
- 7 replies
- 9 kudos
- 9 kudos
Hello @Raj_DB, I took a look at both internal and external documentation and here is what I found. The thread already has the right answer: Lakehouse Federation. @saurabh18cs, @Harshad_B, @AbhilashNagilla, @Aravind_Reddy, @Jen_Evangelista and @balaji...
- 9 kudos
- 247 Views
- 3 replies
- 0 kudos
Resolved! Schema evolution: new NOT NULL column fails on existing Silver data (Lakeflow Connect source)
Hi everyone,We ingest data from our source database with Lakeflow Connect, through a foreign catalog, into a Bronze layer. In the Silver layer, we apply the correct data types and constraints based on our data dictionary.Recently, some tables in the ...
- 247 Views
- 3 replies
- 0 kudos
- 0 kudos
Good points already shared on the nullable → backfill → NOT NULL approach. I'd add a few Lakeflow Connect and Silver recovery considerations, particularly to avoid unnecessary full refreshes.1. Lakeflow Connect: Verify historical data before backfill...
- 0 kudos
- 313 Views
- 5 replies
- 6 kudos
Resolved! How do you collect Excel files from business users into Databricks ?
Hi everyone,I build reports in Databricks from Excel files maintained by many different business users. The files are varied and come from different teams, and I haven’t settled on an approach yet.One idea I’m considering: create a landing location w...
- 313 Views
- 5 replies
- 6 kudos
- 6 kudos
You can use a Unity Catalog Volume backed by S3 as the landing zone and its the right way. You can give each source team their own folder in the volume, allowing users to drop files through the UI. Pipeline can pick up files on arrival using a job fi...
- 6 kudos
- 307 Views
- 3 replies
- 0 kudos
Zerobus support for private-endpoint storage (MLflow traces to Unity Catalog)
We'd like to store MLflow 3 traces in Unity Catalog, using an experiment linked to a UC table-prefix trace location. In our Azure UK South workspace, linking the location fails with:PERMISSION_DENIED: Zerobus failed to open the table. Make sure the t...
- 307 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello @jAAmes_bentley Following up on the two answers above with the doc lines behind them, one Azure change that affects your third question, and an option for now. Thank you for quoting the full error and stating the storage policy up front. That ...
- 0 kudos
- 247 Views
- 3 replies
- 2 kudos
Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation
I'm using Databricks Free Edition and trying to configure an AWS S3 external location through Unity Catalog.My storage credential validation consistently fails at Assume Role, while Self Assume Role and External ID Condition are skipped.Error: Failed...
- 247 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ifra_inam ,The PERMISSION_DENIED failure at the very first Assume Role stage means AWS IAM is instantly rejecting the cross-account trust handshake. Because the failure happens immediately, Databricks never even reaches the S3 permissions check.C...
- 2 kudos
- 105 Views
- 0 replies
- 0 kudos
From the Databricks Hackathon to the Databricks Community!
Back in March 2026, I participated in a Databricks hackathon, where I had the opportunity to build a solution using Databricks and explore its capabilities in solving real-world problems.I'm excited to share that my project secured a place among the ...
- 105 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
