- 138 Views
- 2 replies
- 0 kudos
Creating Databricks agent
Hi All,We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new age...
- 138 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you , we had looked at this but this doesnt help to create an agent from start to finish. Is there any other exercise or example we can look at?
- 0 kudos
- 305 Views
- 3 replies
- 0 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 305 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Aparajita_Nag The "could not be found" error indicates Databricks cannot locate the Access Connector resource in Azure. This happens before any ADLS permissions are checked. Because I lack full information on your setup, you can verify the below ...
- 0 kudos
- 116 Views
- 3 replies
- 1 kudos
S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runtime?
Hi All,If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with ...
- 116 Views
- 3 replies
- 1 kudos
- 1 kudos
@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:What we believed. The pipelines committed every few minutes, so each table's transactio...
- 1 kudos
- 56 Views
- 1 replies
- 1 kudos
Auto SCD API Tombstone Garbage Collection
Are there any settings that can be used to influence the frequency at which the auto SCD API runs the tombstone garbage collection process in Spark Declarative Pipelines? I've seen a couple community posts that referenced the following: pipelines.a...
- 56 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @de01 ,Short answer: Those two settings (pipelines.applyChanges.tombstoneGCFrequencyInSeconds and pipelines.cdc.tombstoneGCFrequencyInSeconds) are internal/undocumented configurations. They are not part of the public API surface and may have been ...
- 1 kudos
- 55 Views
- 1 replies
- 0 kudos
how to enable column level lineage in databricks
I have bronze layer data which is directly referred in silver. there are one or two columns from bronze layer which are used to build silver column.is there any way I can have column-level lineage shown in databricks like table lineage@DATA @lineage
- 55 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Prathamesh, Column level lineage is tracked automatically in Databricks if the tables are governed by Unity Catalog. You do not need to configure explicit column tracking or alter the pipeline code, once your bronze-to-silver transformations run o...
- 0 kudos
- 475 Views
- 6 replies
- 1 kudos
Are We Entering the Context Engineering Era?
I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable...
- 475 Views
- 6 replies
- 1 kudos
- 1 kudos
There is one thing I'd add which is that many large enterprises still struggle with governance, ownership and sometimes even with common definitions of what their data means. The interesting question is not only how much context we can give to tools ...
- 1 kudos
- 249 Views
- 7 replies
- 1 kudos
If statement in DAB YAML file support
Hi,Is there a plan to have if statement support in DAB YAML file?For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow...
- 249 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @QuanDo1 You don't need to use if/else for this use case. Databricks Asset Bundles (DAB) already supports variables that can be overridden per target environment.You can define your variables once and then assign different values for each target (...
- 1 kudos
- 5579 Views
- 4 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 5579 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, this can happen with Excel readers when formula evaluation/cached values are handled differently. In an .xlsx file, the formula itself and its last calculated value are stored separately, so the reader may return the formula string if it doesn't...
- 1 kudos
- 4010 Views
- 3 replies
- 0 kudos
Get information from Power BI via XMLA
Hello everyone I am trying to get information from Power BI semantic models via XMLA endpoint using PySpark in Databricks.Can someone help me with that?tks
- 4010 Views
- 3 replies
- 0 kudos
- 0 kudos
You would need the power bi report to be in a Fabric/premium capacity workspace and Capacity setting XMLA Endpoint = Read (or Read/Write) Any specific reason why via XMLA endpoint?Via the REST API : Datasets - Execute Queries - REST API (Power BI Pow...
- 0 kudos
- 102 Views
- 0 replies
- 0 kudos
Oracle CDC Ingestion Pipeline - Schema Exploration finds 0 tables when DB_DOMAIN
Hi,I'm setting up an Oracle Integrated CDC Managed Ingestion Pipeline (LakeFlow Connect) on Azure Databricks. The pipeline connects to Oracle 19c successfully but Schema Exploration always returns 0 tables.Error:Schema Exploration has COMPLETED in 5 ...
- 102 Views
- 0 replies
- 0 kudos
- 297 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 297 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 264 Views
- 2 replies
- 0 kudos
Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 264 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 202 Views
- 0 replies
- 1 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 202 Views
- 0 replies
- 1 kudos
- 321 Views
- 1 replies
- 2 kudos
Resolved! Lakebase synced table doesn’t recognize Auto CDF on a SDP materialized view
I’m trying to create a triggered Lakebase synced table from a SDP-created materialized view.The source MV uses Automatic CDF: row tracking is enabled and legacy CDF (delta.enableChangeDataFeed) is disabled. The Automatic CDF workspace preview is enab...
- 321 Views
- 1 replies
- 2 kudos
- 2 kudos
Short answer: No — Lakebase synced tables in Triggered/Continuous mode do not currently support Automatic CDF from materialized views. This is a documented limitation, not a configuration issue.The synced tables documentation explicitly states:"Sourc...
- 2 kudos
- 208 Views
- 1 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 208 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read. Is this expected? Not according to anything public. The only documente...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
