- 1539 Views
- 22 replies
- 8 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 1539 Views
- 22 replies
- 8 kudos
- 8 kudos
Hi @dbernstein_tp,Thanks for testing all of that and reporting back. Good to know deletion vectors didn't upset Direct Lake.I think your two findings are really the same problem. If most of the gold MVs are fully recomputed on every update, each refr...
- 8 kudos
- 66 Views
- 1 replies
- 2 kudos
Zerobus to lakebase
Hi,Is there a plan to make lakebase as destination of Zerobus?Thanks.
- 66 Views
- 1 replies
- 2 kudos
- 2 kudos
Did you watch the Bricktalks from last week about zerobus?https://community.databricks.com/t5/bricktalks-tv/recording-bricktalk-zerobus-ingest-amp-cdc-workflows/ba-p/175761Your question came up in the Q&A.Vicky Bukta is the PM and based on what I hea...
- 2 kudos
- 125 Views
- 3 replies
- 0 kudos
Lakeflow Connect (Oracle): backfill a single new column without a full refresh of the table
Hi everyone,We ingest Oracle tables into a Bronze layer with Lakeflow Connect, and our Silver layer is a Lakeflow Spark Declarative Pipeline made of materialized views that read the Bronze tables and apply types and comments from our data dictionary....
- 125 Views
- 3 replies
- 0 kudos
- 0 kudos
Oracle stores the default of a NOT NULL DEFAULT column added this way as metadata. Existing rows aren't physically rewritten, so there's no redo for LogMiner (CDC) and no cursor change for the query-based connector. Either way, the connector correctl...
- 0 kudos
- 120 Views
- 1 replies
- 0 kudos
ASF Corporation
Hello Databricks Community,I'm representing ASF CORPORATION, a piling, foundation engineering and construction company established in 2017 and based in Dhaka, Bangladesh.Our company specializes in hydraulic rotary piling, rotary drilling, auger borin...
- 120 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @asfcorporation see if you can gain some inspiration from here :1) https://www.databricks.com/solutions/accelerators?industry=Manufacturing2)https://www.databricks.com/resources/demos/tutorials/lakehouse-platform/iot-and-predictive-maintenance3) h...
- 0 kudos
- 97 Views
- 2 replies
- 0 kudos
Delta Tables: Is there any possibility on Delta tables Data file formats other than Parquet
Hi Team,Since Parquet files are immutable and Delta tables framework writes data changes into new data files, Is there any option to specify any other file format and try to use same data file for minor data changes.(Below 10 Record transaction chang...
- 97 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @krishgarikipati Could you clarify what specific issue you are facing with the current Parquet-based Delta implementation that makes you want to explore a different file format? Is the concern primarily the amount of data being rewritten for a ver...
- 0 kudos
- 278 Views
- 3 replies
- 2 kudos
guidance on cleainig up after long cdc failure
Hi All,We had a CDC failure which took far to long to resolve. A conversation with vendors on that.Does anyone have any guidance on the process steps to identifying the missed changes and applying them to the tables.I have so far run a query to ident...
- 278 Views
- 3 replies
- 2 kudos
- 2 kudos
Checked the full refresh.I can easily control pauses as we only update end user tables one a dayIt is SQL Server CDC. Full refresh completed Ok
- 2 kudos
- 514 Views
- 5 replies
- 3 kudos
Unity Catalog external location access error after metastore attachment
Hi everyone,I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.Here are the symptoms:The Setup: Storage credentials and an external location were successfully created, and...
- 514 Views
- 5 replies
- 3 kudos
- 3 kudos
This type of issue can sometimes occur when AWS IAM permissions are correctly configured, but Unity Catalog is unable to authorize access through the storage credential or external location. I would recommend checking a few things before changing the...
- 3 kudos
- 833 Views
- 10 replies
- 5 kudos
Resolved! Materialized Views - Strange Difference in Refresh Performance
I have built the same materialized view twice with only one difference: mv01 does incremental refresh and mv02 does full refresh.The mv reads from 7 tables and 1 mv, performs unions, joins, anti-joins as well as filters, and calculates columns on the...
- 833 Views
- 10 replies
- 5 kudos
- 5 kudos
The incremental refresh may actually be doing more work because of the complexity of the query and how changes propagate through joins, unions, and anti-joins. I’d be interested to know whether the query plan shows expensive change detection or recom...
- 5 kudos
- 631 Views
- 7 replies
- 3 kudos
Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves
From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the ...
- 631 Views
- 7 replies
- 3 kudos
- 3 kudos
Hello @sudopradip, I took a look at both internal and external documentation and here is what I found. Credit first. @SumeshKashyap had the practical answer early: write through /Volumes or the Files API from inside Databricks, don’t pin the public I...
- 3 kudos
- 3282 Views
- 5 replies
- 2 kudos
Time Travel Error when selecting from materialized view (Azure Databricks)
Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...
- 3282 Views
- 5 replies
- 2 kudos
- 2 kudos
This is an interesting discussion about handling time travel errors when working with materialized views in Azure Databricks. Understanding how different data objects support historical queries is important, especially when troubleshooting complex da...
- 2 kudos
- 360 Views
- 4 replies
- 0 kudos
Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type
Data Ingestion (Lakeflow) Feature Bug:While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default da...
- 360 Views
- 4 replies
- 0 kudos
- 192 Views
- 2 replies
- 2 kudos
Unable to Attach Databricks Serverless Compute – Environment Version 5 (Heavy Load Warning)
Hi Databricks Community,I'm experiencing an issue with Databricks Serverless Compute while trying to attach it to my notebook using Serverless Environment Version 5.Whenever I attempt to attach the compute, I receive the following warning: "Databrick...
- 192 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @NaniOnyx07, I took a look at both internal and external documentation and here is what I found. First, credit to @ThomazNeto for the thorough first pass. I checked his facts against the docs and he is spot on: environment version 5 shipped in ...
- 2 kudos
- 469 Views
- 7 replies
- 13 kudos
How to Connect Databricks to Snowflake & Query Live Data?
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
- 469 Views
- 7 replies
- 13 kudos
- 13 kudos
Hello @Raj_DB, I took a look at both internal and external documentation and here is what I found. The thread already has the right answer: Lakehouse Federation. @saurabh18cs, @Harshad_B, @AbhilashNagilla, @Aravind_Reddy, @Jen_Evangelista and @balaji...
- 13 kudos
- 297 Views
- 3 replies
- 0 kudos
Resolved! Schema evolution: new NOT NULL column fails on existing Silver data (Lakeflow Connect source)
Hi everyone,We ingest data from our source database with Lakeflow Connect, through a foreign catalog, into a Bronze layer. In the Silver layer, we apply the correct data types and constraints based on our data dictionary.Recently, some tables in the ...
- 297 Views
- 3 replies
- 0 kudos
- 0 kudos
Good points already shared on the nullable → backfill → NOT NULL approach. I'd add a few Lakeflow Connect and Silver recovery considerations, particularly to avoid unnecessary full refreshes.1. Lakeflow Connect: Verify historical data before backfill...
- 0 kudos
- 356 Views
- 5 replies
- 6 kudos
Resolved! How do you collect Excel files from business users into Databricks ?
Hi everyone,I build reports in Databricks from Excel files maintained by many different business users. The files are varied and come from different teams, and I haven’t settled on an approach yet.One idea I’m considering: create a landing location w...
- 356 Views
- 5 replies
- 6 kudos
- 6 kudos
You can use a Unity Catalog Volume backed by S3 as the landing zone and its the right way. You can give each source team their own folder in the volume, allowing users to drop files through the UI. Pipeline can pick up files on arrival using a job fi...
- 6 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
