- 487 Views
- 2 replies
- 2 kudos
Resolved! Recurring Historical Data Modeling Patterns
After reviewing a surprising number of Databricks discussions around SCD2, CDC, historical reporting and temporal joins, I noticed that most historical data modeling challenges seem to fall into a small set of recurring patterns:Historical BackfillLa...
- 487 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello!I would add a few more historical modeling patterns that often appear separately, even though they overlap with SCD2, CDC, or temporal joins.One important case is bi-temporal modeling, where you need to separate business effective time from sys...
- 2 kudos
- 4162 Views
- 1 replies
- 2 kudos
re: Welcoming Bladebridge to Databricks!
Hi @Sujitha and Databricks team,Congrats on the acquisition of Bladebridge. We used this tool a couple years back to migrate an important ETL process from Informatica. I'm glad to see its part of the Data Intelligence Platform and have already take...
- 4162 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi,Thank you for sharing this feedback, and great to hear that you have already used BladeBridge successfully in a previous Informatica migration.I agree that a dedicated BladeBridge forum or community section would be useful, especially now that mor...
- 2 kudos
- 778 Views
- 1 replies
- 1 kudos
New remote (dbfs) caching python library
I had some problems getting much speedup at all from spark or DB disk cache, which I think is essential when developing PySpark code iteratively in notebooks. So I developed a handy caching-library for this which has recently been open sourced, see h...
- 778 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for sharing this. It looks useful, especially for iterative notebook development where the expensive part is not just reading source files but recomputing a complex intermediate DataFrame after several joins or transformations.I can see the va...
- 1 kudos
- 2650 Views
- 4 replies
- 2 kudos
Ingest data from snowflake to databricks
Hi Team,I have some confusion regarding the best approach for ingesting data from Snowflake into Databricks using custom SQL queries.While evaluating the available options, I found multiple approaches:Snowflake Spark ConnectorJDBCQuery FederationLake...
- 2650 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks for raising this question. I recently evaluated similar options for Snowflake-to-Databricks ingestion and would like to share my perspective.From my understanding, the choice depends on whether your primary focus is performance, ease of manage...
- 2 kudos
- 940 Views
- 2 replies
- 0 kudos
AI/BI Genie
I was working with genie and started using Research agent, and now I want to make the genie as a sharable Infrastructure as Code where I can manage all the changes through GitHub and so does anyone have any suggestions how to do this in a best optimi...
- 940 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Gaganmjain_012, I know this is a late reply, but if you are looking for the cleanest way to make a Genie Space shareable and manageable as infrastructure-as-code, the best pattern today is to keep the Genie Space configuration in GitHub and deplo...
- 0 kudos
- 887 Views
- 5 replies
- 2 kudos
Resolved! Declarative pipeline full table refresh generates empty MV.
Hi everyone,- the situation:I have a Declarative pipeline. The pipeline consists several .py files.file1.py: creates a Materialized View: description.file2.py: create 2nd Materialized View by reading a table "transactions" and reading the MV "descri...
- 887 Views
- 5 replies
- 2 kudos
- 2 kudos
@cdn_yyz_yul I think that reading part is where your problem lies. When reading from same pipeline produced datasets, you shouldn't use 3 level name, instead follow DLT's way.Older pipeline versions, where "target" field was used for target schema de...
- 2 kudos
- 704 Views
- 1 replies
- 1 kudos
Resolved! How execute SET spark.sql.sources.partitionOverwriteMode = dynamic; in SQL Stored procedures
Hi,I am able to execute the INSERT OVERWRITE TABLE <tables> PARTITION command , in a notebook cell SET spark.sql.sources.partitionOverwriteMode=dynamic;DECLARE OR REPLACE VARIABLE v_load_date DATE;SET VAR v_load_date = DATE '2026-05-03';INSERT OVERW...
- 704 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @RGSLCA , The short answer is that you can't make SET spark.sql.sources.partitionOverwriteMode=dynamic work from a stored procedure running on a SQL warehouse or serverless. That dynamic partition overwrite path is legacy, and it's SQL-support...
- 1 kudos
- 607 Views
- 1 replies
- 0 kudos
Data bricks + Network security perimeter (storage account) Error
Serverless Compute + NCC configration in account console + Private Endpoint (PE) to storage account is working as expected.When Network security perimeter (NSP) is added and storage account (SA), in NSP we have 2 access mode to choose (transition an...
- 607 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @MyProfile , I looked into your issue and here is what I found: Short version: your instinct is right. The data read over the private endpoint is fine. What's failing is a storage account level call, the Get User Delegation Key operation that U...
- 0 kudos
- 2286 Views
- 2 replies
- 1 kudos
Regarding Databricks Certified Data Engineer Professional recent syllabus change
Dear Team,I am planning to appear Data Engineer Professional certification in June month end. Now I wanted to know will there be any change in syllabus or exam pattern. Because I heard syllabus has been changed for Data Engineer Associate exam.Kind r...
- 2286 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @npalne, I’m not seeing any official update that the Databricks Certified Data Engineer Professional exam has changed. The current Data Engineer Professional certification page still points to the November 30, 2025, exam guide, and the assessment ...
- 1 kudos
- 1344 Views
- 4 replies
- 0 kudos
How to pass Dynamic parameters like job.run_id to a pipeline_task in Databricks workflow job?
Is it possible to pass dynamic parameters, such as job.run_id to a pipeline_task within a Databricks Workflow job?
- 1344 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @NancyX ,In addition to @Ashwin_DSA 's explanation, if you need to do the same from Declarative Automation Bundles (DABs formerly) workflow YAML markup, here are a couple examples from one of my jobs.Run ID:- name: job_run_id default: "{{job...
- 0 kudos
- 662 Views
- 2 replies
- 1 kudos
Sync Tables: Unity Catalog to Lakebase - Materialized Views triggered mode
Hi,I am facing some issues syncing Materialized Views into Lakebase with triggered mode, and I actually wonder if this is actually possible.create materialized view some_catalog.some_schema.some_table tblproperties( 'delta.enableChangeDataFeed' = '...
- 662 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, For the CDC on Materialised views, as the feature is in private preview at the moment you would need to speak to your account team to get it enabled. You also need to be using DBR 18.1 or above on classic compute for it to work. The workaround wo...
- 1 kudos
- 1203 Views
- 2 replies
- 0 kudos
Lakeflow Connect - Column filtering
Hi community , I am interested in learning more about the feature that was mentioned in recent summit about query pushdown in lakeflow connect for SQL server. I believe this feature will allow to select only the required columns from source tables. I...
- 1203 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @amitpm, Just closing the loop on this thread, and apologies for the very late follow-up. The feature being discussed here maps to Lakeflow Connect query-based connectors for SQL Server, and that capability is now available. Query-based connectors...
- 0 kudos
- 422 Views
- 2 replies
- 1 kudos
Resolved! MongoDB Spark Connector UC command is not supported without recommendation
I am trying to read a MongoDB collection using spark.read.format("mongodb"). However, when I attempt to display the collection, I receive the error: "UC command is not supported without recommendation." Please help resolve this issue.
- 422 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @shan-databricks, This is expected behaviour on Unity Catalog shared or standard compute. The mongodb Spark data source path can fail there with UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION / SQLSTATE 0AKUC. The public docs for that are UC_COMM...
- 1 kudos
- 2865 Views
- 2 replies
- 4 kudos
Delta Sharing Costs
When Delta Sharing is enabled and a link is shared, I understand that the data transfer happens directly and not through the sharing server. I'm curious how costs are calculated. Is the entity making the share available charged for data egress and ...
- 2865 Views
- 2 replies
- 4 kudos
- 4 kudos
I am also curious about this - is there anyway to monitor this apart from system.billing.usage table?
- 4 kudos
- 630 Views
- 3 replies
- 1 kudos
Resolved! Find and replace not saving changes in Databricks Notebook problem
In Databricks notebooks, the find and replace functionality doesn't appear to be saving the replaced text. When I use find and replace text, it does initially appear to work correctly (I see that the highlighted texts are replaced).However, when I sc...
- 630 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @milan2, Thanks for raising this. What you are describing sounds like the text is being visually replaced in the editor, but the change is not actually being persisted. We have also seen a recent report of similar behaviour where using the browser...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
