- 107 Views
- 6 replies
- 3 kudos
Missing spark structured streaming metadata blobs?
I'm exploring spark structured streaming, and contemplating uses for it.I'd like to launch a job on a recurring basis from an external scheduler. It will launch a spark job with trigger (availableNow=True). I'm concerned about overlapping executions...
- 107 Views
- 6 replies
- 3 kudos
- 3 kudos
Hi @DB1To3 Just make this concurrent run to 1 for your streaming jobs are your problem is sorted , it wont corrupt checkpoint locatiion
- 3 kudos
- 133 Views
- 3 replies
- 2 kudos
Zerobus to lakebase
Hi,Is there a plan to make lakebase as destination of Zerobus?Thanks.
- 133 Views
- 3 replies
- 2 kudos
- 2 kudos
Thanks for the link. IMHO, it is the opposite. Previously when there is no option, we have to load data from Kafka to delta table. Now with Lakebase and LTAP, we can consider to use Lakebase as a fast landing zone. Instead of loading data to Lakehous...
- 2 kudos
- 71 Views
- 0 replies
- 0 kudos
Making AI-agent writes safe on Databricks: lineage, shallow clones, null-safe diffs, rollback
A pipeline change can run successfully and still ship wrong data: a join that duplicates rows, a filter that moves a revenue total by a few percent. With AI agents now writing more pipeline code, I wanted a check that runs before a change reaches pro...
- 71 Views
- 0 replies
- 0 kudos
- 185 Views
- 2 replies
- 0 kudos
ASF Corporation
Hello Databricks Community,I'm representing ASF CORPORATION, a piling, foundation engineering and construction company established in 2017 and based in Dhaka, Bangladesh.Our company specializes in hydraulic rotary piling, rotary drilling, auger borin...
- 185 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @asfcorporation, and welcome to the Databricks Community!Beyond the usual Lakeflow + Unity Catalog + AI/BI Dashboards stack, I’d suggest not inventing your own schema: geotechnical data already has open standards. AGS4 (UK) and DIGGS (US, by ASCE ...
- 0 kudos
- 579 Views
- 8 replies
- 14 kudos
How to Connect Databricks to Snowflake & Query Live Data?
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
- 579 Views
- 8 replies
- 14 kudos
- 14 kudos
Hello @Raj_DB, I took a look at both internal and external documentation and here is what I found. The thread already has the right answer: Lakehouse Federation. @saurabh18cs, @Harshad_B, @AbhilashNagilla, @Aravind_Reddy, @Jen_Evangelista and @balaji...
- 14 kudos
- 1662 Views
- 22 replies
- 8 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 1662 Views
- 22 replies
- 8 kudos
- 8 kudos
Hi @dbernstein_tp,Thanks for testing all of that and reporting back. Good to know deletion vectors didn't upset Direct Lake.I think your two findings are really the same problem. If most of the gold MVs are fully recomputed on every update, each refr...
- 8 kudos
- 185 Views
- 3 replies
- 0 kudos
Lakeflow Connect (Oracle): backfill a single new column without a full refresh of the table
Hi everyone,We ingest Oracle tables into a Bronze layer with Lakeflow Connect, and our Silver layer is a Lakeflow Spark Declarative Pipeline made of materialized views that read the Bronze tables and apply types and comments from our data dictionary....
- 185 Views
- 3 replies
- 0 kudos
- 0 kudos
Oracle stores the default of a NOT NULL DEFAULT column added this way as metadata. Existing rows aren't physically rewritten, so there's no redo for LogMiner (CDC) and no cursor change for the query-based connector. Either way, the connector correctl...
- 0 kudos
- 158 Views
- 2 replies
- 0 kudos
Delta Tables: Is there any possibility on Delta tables Data file formats other than Parquet
Hi Team,Since Parquet files are immutable and Delta tables framework writes data changes into new data files, Is there any option to specify any other file format and try to use same data file for minor data changes.(Below 10 Record transaction chang...
- 158 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @krishgarikipati Could you clarify what specific issue you are facing with the current Parquet-based Delta implementation that makes you want to explore a different file format? Is the concern primarily the amount of data being rewritten for a ver...
- 0 kudos
- 319 Views
- 3 replies
- 2 kudos
guidance on cleainig up after long cdc failure
Hi All,We had a CDC failure which took far to long to resolve. A conversation with vendors on that.Does anyone have any guidance on the process steps to identifying the missed changes and applying them to the tables.I have so far run a query to ident...
- 319 Views
- 3 replies
- 2 kudos
- 2 kudos
Checked the full refresh.I can easily control pauses as we only update end user tables one a dayIt is SQL Server CDC. Full refresh completed Ok
- 2 kudos
- 571 Views
- 5 replies
- 3 kudos
Unity Catalog external location access error after metastore attachment
Hi everyone,I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.Here are the symptoms:The Setup: Storage credentials and an external location were successfully created, and...
- 571 Views
- 5 replies
- 3 kudos
- 3 kudos
This type of issue can sometimes occur when AWS IAM permissions are correctly configured, but Unity Catalog is unable to authorize access through the storage credential or external location. I would recommend checking a few things before changing the...
- 3 kudos
- 929 Views
- 10 replies
- 5 kudos
Resolved! Materialized Views - Strange Difference in Refresh Performance
I have built the same materialized view twice with only one difference: mv01 does incremental refresh and mv02 does full refresh.The mv reads from 7 tables and 1 mv, performs unions, joins, anti-joins as well as filters, and calculates columns on the...
- 929 Views
- 10 replies
- 5 kudos
- 5 kudos
The incremental refresh may actually be doing more work because of the complexity of the query and how changes propagate through joins, unions, and anti-joins. I’d be interested to know whether the query plan shows expensive change detection or recom...
- 5 kudos
- 682 Views
- 7 replies
- 3 kudos
Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves
From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the ...
- 682 Views
- 7 replies
- 3 kudos
- 3 kudos
Hello @sudopradip, I took a look at both internal and external documentation and here is what I found. Credit first. @SumeshKashyap had the practical answer early: write through /Volumes or the Files API from inside Databricks, don’t pin the public I...
- 3 kudos
- 3324 Views
- 5 replies
- 2 kudos
Time Travel Error when selecting from materialized view (Azure Databricks)
Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...
- 3324 Views
- 5 replies
- 2 kudos
- 2 kudos
This is an interesting discussion about handling time travel errors when working with materialized views in Azure Databricks. Understanding how different data objects support historical queries is important, especially when troubleshooting complex da...
- 2 kudos
- 411 Views
- 4 replies
- 0 kudos
Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type
Data Ingestion (Lakeflow) Feature Bug:While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default da...
- 411 Views
- 4 replies
- 0 kudos
- 244 Views
- 2 replies
- 2 kudos
Unable to Attach Databricks Serverless Compute – Environment Version 5 (Heavy Load Warning)
Hi Databricks Community,I'm experiencing an issue with Databricks Serverless Compute while trying to attach it to my notebook using Serverless Environment Version 5.Whenever I attempt to attach the compute, I receive the following warning: "Databrick...
- 244 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @NaniOnyx07, I took a look at both internal and external documentation and here is what I found. First, credit to @ThomazNeto for the thorough first pass. I checked his facts against the docs and he is spot on: environment version 5 shipped in ...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
