- 63 Views
- 1 replies
- 1 kudos
S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runtime?
Hi All,If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with ...
- 63 Views
- 1 replies
- 1 kudos
- 1 kudos
check whether the latest Databricks runtime has an equivalent S3 listing optimization. If not, catalog-managed commits or reducing commit frequency/retention seem to be the only practical ways to control the request costs without much complexity.
- 1 kudos
- 408 Views
- 6 replies
- 1 kudos
Are We Entering the Context Engineering Era?
I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable...
- 408 Views
- 6 replies
- 1 kudos
- 1 kudos
There is one thing I'd add which is that many large enterprises still struggle with governance, ownership and sometimes even with common definitions of what their data means. The interesting question is not only how much context we can give to tools ...
- 1 kudos
- 184 Views
- 7 replies
- 1 kudos
If statement in DAB YAML file support
Hi,Is there a plan to have if statement support in DAB YAML file?For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow...
- 184 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @QuanDo1 You don't need to use if/else for this use case. Databricks Asset Bundles (DAB) already supports variables that can be overridden per target environment.You can define your variables once and then assign different values for each target (...
- 1 kudos
- 5559 Views
- 4 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 5559 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, this can happen with Excel readers when formula evaluation/cached values are handled differently. In an .xlsx file, the formula itself and its last calculated value are stored separately, so the reader may return the formula string if it doesn't...
- 1 kudos
- 4005 Views
- 3 replies
- 0 kudos
Get information from Power BI via XMLA
Hello everyone I am trying to get information from Power BI semantic models via XMLA endpoint using PySpark in Databricks.Can someone help me with that?tks
- 4005 Views
- 3 replies
- 0 kudos
- 0 kudos
You would need the power bi report to be in a Fabric/premium capacity workspace and Capacity setting XMLA Endpoint = Read (or Read/Write) Any specific reason why via XMLA endpoint?Via the REST API : Datasets - Execute Queries - REST API (Power BI Pow...
- 0 kudos
- 92 Views
- 0 replies
- 0 kudos
Oracle CDC Ingestion Pipeline - Schema Exploration finds 0 tables when DB_DOMAIN
Hi,I'm setting up an Oracle Integrated CDC Managed Ingestion Pipeline (LakeFlow Connect) on Azure Databricks. The pipeline connects to Oracle 19c successfully but Schema Exploration always returns 0 tables.Error:Schema Exploration has COMPLETED in 5 ...
- 92 Views
- 0 replies
- 0 kudos
- 289 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 289 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 250 Views
- 2 replies
- 0 kudos
Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 250 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 193 Views
- 0 replies
- 1 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 193 Views
- 0 replies
- 1 kudos
- 301 Views
- 1 replies
- 1 kudos
Resolved! Lakebase synced table doesn’t recognize Auto CDF on a SDP materialized view
I’m trying to create a triggered Lakebase synced table from a SDP-created materialized view.The source MV uses Automatic CDF: row tracking is enabled and legacy CDF (delta.enableChangeDataFeed) is disabled. The Automatic CDF workspace preview is enab...
- 301 Views
- 1 replies
- 1 kudos
- 1 kudos
Short answer: No — Lakebase synced tables in Triggered/Continuous mode do not currently support Automatic CDF from materialized views. This is a documented limitation, not a configuration issue.The synced tables documentation explicitly states:"Sourc...
- 1 kudos
- 203 Views
- 1 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 203 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read. Is this expected? Not according to anything public. The only documente...
- 0 kudos
- 21622 Views
- 4 replies
- 2 kudos
Resolved! ThreadPoolExecutor in Databricks
I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...
- 21622 Views
- 4 replies
- 2 kudos
- 2 kudos
Good point on the threading vs. process distinction. To add a bit morecontext:Root cause: ThreadPoolExecutor only parallelizes within the driver'ssingle Python process. All threads share the same driver memory, sopandas DataFrame creation across thre...
- 2 kudos
- 3168 Views
- 3 replies
- 2 kudos
Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most...
- 3168 Views
- 3 replies
- 2 kudos
- 2 kudos
The executor heartbeat timeout makes sense to investigate first, especially with a table that has grown to around 1.5 crore rows. I’d be interested to know whether increasing executor resources actually resolved the issue before changing the command ...
- 2 kudos
- 301 Views
- 2 replies
- 3 kudos
Resolved! Data quality Lineage Root cause analysis
Hi everyone,I'm working on a client use case where they want to visualize data quality together with data lineage.My current approach is to combine:Unity Catalog Data Lineage for upstream/downstream dependenciesLakehouse Monitoring (or other data mon...
- 301 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Niyojit ,I think your proposed architecture is broadly the right direction, but I would separate lineage collection from lineage visualization.Unity Catalog already exposes table-to-table relationships programmatically through system.access.tabl...
- 3 kudos
- 351 Views
- 3 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 351 Views
- 3 replies
- 0 kudos
- 0 kudos
You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.Silver LayerNew Columns - You can l...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
