- 637 Views
- 3 replies
- 0 kudos
Resolved! Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 637 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , glad the first answer helped. Your sliding window observation is sharp, but one correction to the mental model: delta.deletedFileRetentionDuration only drives physical deletion at the moment a VACUUM executes. Between your w...
- 0 kudos
- 751 Views
- 1 replies
- 0 kudos
Resolved! Databricks delta tables and Iceberg
Hi experts ,now databricks for support for Iceberg open table format conversion yet not fully though.Databricks calls this Delta Lake UniForm (Universal Format). When tentatively Databricks can support fully open table format Iceberg, Hudi etc.Is the...
- 751 Views
- 1 replies
- 0 kudos
- 0 kudos
Worth separating "reading Iceberg from Delta" from "Databricks fully adopting Iceberg as a first-class table format" - those are two different things and Databricks has been moving on both fronts. Today: UniForm (Universal Format) makes a Delta table...
- 0 kudos
- 262 Views
- 1 replies
- 0 kudos
transformWithState might be causing 'Module not found'
Hi,So i've been playing around with transformWithState over the last few weeks and have got the code to work exactly as desired.Wrote some integration tests that execute fine within the workspace in serverless and everything, the issue is however is ...
- 262 Views
- 1 replies
- 0 kudos
- 0 kudos
This smells like a client/server module-resolution mismatch that's specific to Databricks Connect + serverless, rather than a wheel packaging problem per se. A few things worth checking: 1. Where the StatefulProcessor class lives. Since it's defined ...
- 0 kudos
- 186 Views
- 0 replies
- 1 kudos
Databricks Certification Exam Suspended - Databricks Certified Data Engineer Associate
Dear Databricks Certification Support Team,My Databricks Certified Data Engineer Associate exam was suspended during the exam session.After the exam was suspended, I verified my system and found that my webcam was not working properly. I believe the ...
- 186 Views
- 0 replies
- 1 kudos
- 1320 Views
- 6 replies
- 2 kudos
Code on cluster runs idefinitely
Hello.Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:- Cluster starts, have running status- I run code, for example print("Hello")- Code runs indefinitely- I click interrupt, it st...
- 1320 Views
- 6 replies
- 2 kudos
- 2 kudos
It sounds more like a cluster/driver initialization or resource issue than a Python problem, especially because even SELECT 1 hangs.I’d check these first:Look at the Driver logs and Event log for errors around the first notebook execution. Databricks...
- 2 kudos
- 467 Views
- 4 replies
- 0 kudos
Creating Databricks agent
Hi All,We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new age...
- 467 Views
- 4 replies
- 0 kudos
- 0 kudos
If you're approaching this specifically as a migration validation use case, I'd separate the agent into two layers: the validation tools and the agent reasoning layer.For example, expose controlled tools for querying Hive and Databricks, then have th...
- 0 kudos
- 12623 Views
- 10 replies
- 1 kudos
Error when reading delta lake files with Auto Loader
Hi,When reading Delta Lake file (created by Auto Loader) with this code: df = ( spark.readStream .format('cloudFiles') .option("cloudFiles.format", "delta") .option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint") .load(bronz...
- 12623 Views
- 10 replies
- 1 kudos
- 1 kudos
The key point is using the correct format for Delta tables rather than treating them like raw files. The same idea of using the right tool for the job applies in other technical workflows too, such as a delta executor apk when handling Android relate...
- 1 kudos
- 402 Views
- 2 replies
- 0 kudos
how to enable column level lineage in databricks
I have bronze layer data which is directly referred in silver. there are one or two columns from bronze layer which are used to build silver column.is there any way I can have column-level lineage shown in databricks like table lineage@DATA @lineage
- 402 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @prathamesh1982 , Good news: there's nothing to enable. @balajij8 has it right, Unity Catalog captures column-level lineage automatically. I'll add some prerequisites and gotchas to check if the lineage doesn't appear. If your bronze and silver t...
- 0 kudos
- 400 Views
- 2 replies
- 1 kudos
S3 LIST costs on high commit rate Delta tables: is there a start-after option on Databricks runtime?
Hi All,If you run streaming or continuous pipelines that commit to Delta tables on S3 every few minutes, your _delta_log folders are probably much bigger than your tables. A CDC or MERGE flow commits every cycle whether or not data arrived, and with ...
- 400 Views
- 2 replies
- 1 kudos
- 1 kudos
@brianjorge178 @cartergray70543 Thanks for your thoughts. I had a long chat with Databricks support team, and here is where have landed so far if it interests you:What we believed. The pipelines committed every few minutes, so each table's transactio...
- 1 kudos
- 1341 Views
- 6 replies
- 1 kudos
Are We Entering the Context Engineering Era?
I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable...
- 1341 Views
- 6 replies
- 1 kudos
- 1 kudos
There is one thing I'd add which is that many large enterprises still struggle with governance, ownership and sometimes even with common definitions of what their data means. The interesting question is not only how much context we can give to tools ...
- 1 kudos
- 710 Views
- 7 replies
- 1 kudos
If statement in DAB YAML file support
Hi,Is there a plan to have if statement support in DAB YAML file?For example, I would like to have different schedule for jobs based on the environment (higher frequency in PROD, lower frequency in DEV, QA). How can I do it in Databricks via workflow...
- 710 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi @QuanDo1 You don't need to use if/else for this use case. Databricks Asset Bundles (DAB) already supports variables that can be overridden per target environment.You can define your variables once and then assign different values for each target (...
- 1 kudos
- 5933 Views
- 4 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 5933 Views
- 4 replies
- 1 kudos
- 1 kudos
Yes, this can happen with Excel readers when formula evaluation/cached values are handled differently. In an .xlsx file, the formula itself and its last calculated value are stored separately, so the reader may return the formula string if it doesn't...
- 1 kudos
- 4144 Views
- 3 replies
- 0 kudos
Get information from Power BI via XMLA
Hello everyone I am trying to get information from Power BI semantic models via XMLA endpoint using PySpark in Databricks.Can someone help me with that?tks
- 4144 Views
- 3 replies
- 0 kudos
- 0 kudos
You would need the power bi report to be in a Fabric/premium capacity workspace and Capacity setting XMLA Endpoint = Read (or Read/Write) Any specific reason why via XMLA endpoint?Via the REST API : Datasets - Execute Queries - REST API (Power BI Pow...
- 0 kudos
- 551 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 551 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 689 Views
- 2 replies
- 0 kudos
Resolved! Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 689 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
