- 638 Views
- 2 replies
- 1 kudos
S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runtime?
Hi All,If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with ...
- 638 Views
- 2 replies
- 1 kudos
- 1 kudos
@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:What we believed. The pipelines committed every few minutes, so each table's transactio...
- 1 kudos
- 2013 Views
- 6 replies
- 1 kudos
Are We Entering the Context Engineering Era?
I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable...
- 2013 Views
- 6 replies
- 1 kudos
- 1 kudos
There is one thing I'd add which is that many large enterprises still struggle with governance, ownership and sometimes even with common definitions of what their data means. The interesting question is not only how much context we can give to tools ...
- 1 kudos
- 1383 Views
- 7 replies
- 1 kudos
If statement in DAB YAML file support
Hi,Is there a plan to have if statement support in DAB YAML file?For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow...
- 1383 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @QuanDo1 You don't need to use if/else for this use case. Databricks Asset Bundles (DAB) already supports variables that can be overridden per target environment.You can define your variables once and then assign different values for each target (...
- 1 kudos
- 6622 Views
- 4 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 6622 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, this can happen with Excel readers when formula evaluation/cached values are handled differently. In an .xlsx file, the formula itself and its last calculated value are stored separately, so the reader may return the formula string if it doesn't...
- 1 kudos
- 4321 Views
- 3 replies
- 0 kudos
Get information from Power BI via XMLA
Hello everyone I am trying to get information from Power BI semantic models via XMLA endpoint using PySpark in Databricks.Can someone help me with that?tks
- 4321 Views
- 3 replies
- 0 kudos
- 0 kudos
You would need the power bi report to be in a Fabric/premium capacity workspace and Capacity setting XMLA Endpoint = Read (or Read/Write) Any specific reason why via XMLA endpoint?Via the REST API : Datasets - Execute Queries - REST API (Power BI Pow...
- 0 kudos
- 958 Views
- 2 replies
- 0 kudos
Resolved! Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 958 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 460 Views
- 1 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 460 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read. Is this expected? Not according to anything public. The only documente...
- 0 kudos
- 22440 Views
- 4 replies
- 2 kudos
Resolved! ThreadPoolExecutor in Databricks
I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...
- 22440 Views
- 4 replies
- 2 kudos
- 2 kudos
Good point on the threading vs. process distinction. To add a bit morecontext:Root cause: ThreadPoolExecutor only parallelizes within the driver'ssingle Python process. All threads share the same driver memory, sopandas DataFrame creation across thre...
- 2 kudos
- 3539 Views
- 3 replies
- 2 kudos
Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most...
- 3539 Views
- 3 replies
- 2 kudos
- 2 kudos
The executor heartbeat timeout makes sense to investigate first, especially with a table that has grown to around 1.5 crore rows. I’d be interested to know whether increasing executor resources actually resolved the issue before changing the command ...
- 2 kudos
- 1088 Views
- 2 replies
- 3 kudos
Resolved! Data quality Lineage Root cause analysis
Hi everyone,I'm working on a client use case where they want to visualize data quality together with data lineage.My current approach is to combine:Unity Catalog Data Lineage for upstream/downstream dependenciesLakehouse Monitoring (or other data mon...
- 1088 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Niyojit ,I think your proposed architecture is broadly the right direction, but I would separate lineage collection from lineage visualization.Unity Catalog already exposes table-to-table relationships programmatically through system.access.tabl...
- 3 kudos
- 975 Views
- 3 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 975 Views
- 3 replies
- 0 kudos
- 0 kudos
You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.Silver LayerNew Columns - You can l...
- 0 kudos
- 1015 Views
- 3 replies
- 0 kudos
Resolved! Skipping malformed records when reading Avro-files
Using read_files to read Avro-formatted blobs from an external location, I get the error:> Malformed records are detected in record parsing: Event 0. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as ...
- 1015 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for the workaround!That said, I do think this is if nothing else, a documentation issue with Databricks.Documentation says:> For options specific to each file format (JSON, CSV, XML, Parquet, Avro, text, ORC, and binary), see DataFrameReader o...
- 0 kudos
- 2870 Views
- 6 replies
- 17 kudos
Resolved! Why is the default auto-termination for serverless interactive notebook compute 60 minutes?
We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than act...
- 2870 Views
- 6 replies
- 17 kudos
- 17 kudos
Hi @elizeu_reis , great detective work here. Cross-referencing system.billing.usage with system.query.history is exactly the right way to separate real usage from idle burn. I did some digging, and I can confirm most of what you observed. There are t...
- 17 kudos
- 710 Views
- 3 replies
- 0 kudos
Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow
Hi All,I want to know my lakeflow connect pipeline has stopped running because Change Tracking is enabled and table doesn't have a primary key and i want to switch to CDC now. Will it impact my pipeline. What are the steps i need to follow to on inge...
- 710 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Upendra,When both Change Tracking and Change Data Capture are enabled the connector generally prioritizes Change Tracking. Since CT requires the source table to have a primary key and table did not have one, it causes the pipeline to fail. Even af...
- 0 kudos
- 5002 Views
- 3 replies
- 2 kudos
Pass variable from one notebook to another
I have a main notebook where I am setting a python variable to the name of the catalog I want to work in. I then call another notebook, using %run, which runs an insert into using a SQL command where I want to specify the catalog using the catalog v...
- 5002 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @ADB0513 With %run, you dont need to pass anything. %run runs the child inline in the same Python namespace, so the catalog variable set in the main notebook is already visible in the child. Just use it. # main catalog = "dev_catalog" # %run ./chi...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
