- 881 Views
- 1 replies
- 1 kudos
Resolved! Databricks apps
I have multiple Databricks Apps running, but their usage is not fixed or predictable. Some apps are used only occasionally, while others may remain idle for long periods.Since Databricks Apps need to stay up and continue consuming resources even when...
- 881 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @anmolhhns, I couldn't find any public documentation showing that Databricks Apps supports automatic idle shutdown or usage-based scale-to-zero for the app runtime itself. The current documented lifecycle is that an app can be Running, Stopped, De...
- 1 kudos
- 426 Views
- 3 replies
- 0 kudos
[Lakeflow Spark Declarative Pipelines] - Compatibility Mode not working
I’m working with an SDP pipeline that creates a streaming table using the dlt.create_streaming_table decorator. My goal is to expose this table through an external location so that a client can read it from Snowflake.I attempted to configure this dir...
- 426 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Jotaefe1991 ,The overlap error you are hitting is a Unity Catalog storage collision, not a DLT limitation.Here is exactly what is happening and how to fix it:The path you provided for "delta.universalFormat.compatibility.location" (abfss://.../br...
- 0 kudos
- 487 Views
- 2 replies
- 2 kudos
Resolved! Recurring Historical Data Modeling Patterns
After reviewing a surprising number of Databricks discussions around SCD2, CDC, historical reporting and temporal joins, I noticed that most historical data modeling challenges seem to fall into a small set of recurring patterns:Historical BackfillLa...
- 487 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello!I would add a few more historical modeling patterns that often appear separately, even though they overlap with SCD2, CDC, or temporal joins.One important case is bi-temporal modeling, where you need to separate business effective time from sys...
- 2 kudos
- 4163 Views
- 1 replies
- 2 kudos
re: Welcoming Bladebridge to Databricks!
Hi @Sujitha and Databricks team,Congrats on the acquisition of Bladebridge. We used this tool a couple years back to migrate an important ETL process from Informatica. I'm glad to see its part of the Data Intelligence Platform and have already take...
- 4163 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi,Thank you for sharing this feedback, and great to hear that you have already used BladeBridge successfully in a previous Informatica migration.I agree that a dedicated BladeBridge forum or community section would be useful, especially now that mor...
- 2 kudos
- 780 Views
- 1 replies
- 1 kudos
New remote (dbfs) caching python library
I had some problems getting much speedup at all from spark or DB disk cache, which I think is essential when developing PySpark code iteratively in notebooks. So I developed a handy caching-library for this which has recently been open sourced, see h...
- 780 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for sharing this. It looks useful, especially for iterative notebook development where the expensive part is not just reading source files but recomputing a complex intermediate DataFrame after several joins or transformations.I can see the va...
- 1 kudos
- 2673 Views
- 4 replies
- 2 kudos
Ingest data from snowflake to databricks
Hi Team,I have some confusion regarding the best approach for ingesting data from Snowflake into Databricks using custom SQL queries.While evaluating the available options, I found multiple approaches:Snowflake Spark ConnectorJDBCQuery FederationLake...
- 2673 Views
- 4 replies
- 2 kudos
- 2 kudos
Thanks for raising this question. I recently evaluated similar options for Snowflake-to-Databricks ingestion and would like to share my perspective.From my understanding, the choice depends on whether your primary focus is performance, ease of manage...
- 2 kudos
- 942 Views
- 2 replies
- 0 kudos
AI/BI Genie
I was working with genie and started using Research agent, and now I want to make the genie as a sharable Infrastructure as Code where I can manage all the changes through GitHub and so does anyone have any suggestions how to do this in a best optimi...
- 942 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Gaganmjain_012, I know this is a late reply, but if you are looking for the cleanest way to make a Genie Space shareable and manageable as infrastructure-as-code, the best pattern today is to keep the Genie Space configuration in GitHub and deplo...
- 0 kudos
- 890 Views
- 5 replies
- 2 kudos
Resolved! Declarative pipeline full table refresh generates empty MV.
Hi everyone,- the situation:I have a Declarative pipeline. The pipeline consists several .py files.file1.py: creates a Materialized View: description.file2.py: create 2nd Materialized View by reading a table "transactions" and reading the MV "descri...
- 890 Views
- 5 replies
- 2 kudos
- 2 kudos
@cdn_yyz_yul I think that reading part is where your problem lies. When reading from same pipeline produced datasets, you shouldn't use 3 level name, instead follow DLT's way.Older pipeline versions, where "target" field was used for target schema de...
- 2 kudos
- 708 Views
- 1 replies
- 1 kudos
Resolved! How execute SET spark.sql.sources.partitionOverwriteMode = dynamic; in SQL Stored procedures
Hi,I am able to execute the INSERT OVERWRITE TABLE <tables> PARTITION command , in a notebook cell SET spark.sql.sources.partitionOverwriteMode=dynamic;DECLARE OR REPLACE VARIABLE v_load_date DATE;SET VAR v_load_date = DATE '2026-05-03';INSERT OVERW...
- 708 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @RGSLCA , The short answer is that you can't make SET spark.sql.sources.partitionOverwriteMode=dynamic work from a stored procedure running on a SQL warehouse or serverless. That dynamic partition overwrite path is legacy, and it's SQL-support...
- 1 kudos
- 610 Views
- 1 replies
- 0 kudos
Data bricks + Network security perimeter (storage account) Error
Serverless Compute + NCC configration in account console + Private Endpoint (PE) to storage account is working as expected.When Network security perimeter (NSP) is added and storage account (SA), in NSP we have 2 access mode to choose (transition an...
- 610 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @MyProfile , I looked into your issue and here is what I found: Short version: your instinct is right. The data read over the private endpoint is fine. What's failing is a storage account level call, the Get User Delegation Key operation that U...
- 0 kudos
- 2299 Views
- 2 replies
- 1 kudos
Regarding Databricks Certified Data Engineer Professional recent syllabus change
Dear Team,I am planning to appear Data Engineer Professional certification in June month end. Now I wanted to know will there be any change in syllabus or exam pattern. Because I heard syllabus has been changed for Data Engineer Associate exam.Kind r...
- 2299 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @npalne, I’m not seeing any official update that the Databricks Certified Data Engineer Professional exam has changed. The current Data Engineer Professional certification page still points to the November 30, 2025, exam guide, and the assessment ...
- 1 kudos
- 1344 Views
- 4 replies
- 0 kudos
How to pass Dynamic parameters like job.run_id to a pipeline_task in Databricks workflow job?
Is it possible to pass dynamic parameters, such as job.run_id to a pipeline_task within a Databricks Workflow job?
- 1344 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @NancyX ,In addition to @Ashwin_DSA 's explanation, if you need to do the same from Declarative Automation Bundles (DABs formerly) workflow YAML markup, here are a couple examples from one of my jobs.Run ID:- name: job_run_id default: "{{job...
- 0 kudos
- 664 Views
- 2 replies
- 1 kudos
Sync Tables: Unity Catalog to Lakebase - Materialized Views triggered mode
Hi,I am facing some issues syncing Materialized Views into Lakebase with triggered mode, and I actually wonder if this is actually possible.create materialized view some_catalog.some_schema.some_table tblproperties( 'delta.enableChangeDataFeed' = '...
- 664 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, For the CDC on Materialised views, as the feature is in private preview at the moment you would need to speak to your account team to get it enabled. You also need to be using DBR 18.1 or above on classic compute for it to work. The workaround wo...
- 1 kudos
- 1205 Views
- 2 replies
- 0 kudos
Lakeflow Connect - Column filtering
Hi community , I am interested in learning more about the feature that was mentioned in recent summit about query pushdown in lakeflow connect for SQL server. I believe this feature will allow to select only the required columns from source tables. I...
- 1205 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @amitpm, Just closing the loop on this thread, and apologies for the very late follow-up. The feature being discussed here maps to Lakeflow Connect query-based connectors for SQL Server, and that capability is now available. Query-based connectors...
- 0 kudos
- 422 Views
- 2 replies
- 1 kudos
Resolved! MongoDB Spark Connector UC command is not supported without recommendation
I am trying to read a MongoDB collection using spark.read.format("mongodb"). However, when I attempt to display the collection, I receive the error: "UC command is not supported without recommendation." Please help resolve this issue.
- 422 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @shan-databricks, This is expected behaviour on Unity Catalog shared or standard compute. The mongodb Spark data source path can fail there with UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION / SQLSTATE 0AKUC. The public docs for that are UC_COMM...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
