- 72 Views
- 1 replies
- 0 kudos
Transaction log integrity issue
Delta transaction log for one of our tables which is being written to by a Kafka Connect IcebergSinkConnector via the Unity Catalog Iceberg REST endpoint, is currently corrupted and is failing when trying to read with the below exceptionERROR:com.dat...
- 72 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @analyticsnerd , Delta maintains a checksum file (.crc) for each committed version, along with the commit log (.json), recording the table’s file count, total size, and a distribution of file sizes (a histogram that buckets files by size). For ...
- 0 kudos
- 257 Views
- 2 replies
- 1 kudos
Cross-engine Iceberg V3 compatibility issue- Databricks DELETE deletion vectors not readable by ext
Hi Databricks Support Team,We are raising this ticket to report a cross-engine compatibility issue between Databricks and Teradata when performing DELETE operations on an Iceberg V3 table.---ENVIRONMENT DETAILS-------------------Databricks Runtime V...
- 257 Views
- 2 replies
- 1 kudos
- 1 kudos
This appears to be a cross-engine Iceberg V3 compatibility issue. If Databricks writes .bin deletion vectors while Teradata only supports Puffin, the key question is whether Puffin output can be enabled in DBR 17.3. I’d ask Databricks to confirm curr...
- 1 kudos
- 208 Views
- 6 replies
- 1 kudos
Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 208 Views
- 6 replies
- 1 kudos
- 1 kudos
Use a single Auto Loader stream and extract the objecttype from _metadata.file_name. Then route each batch to the appropriate table based on that value. This scales much better than repeatedly listing files and creating separate streams for each obje...
- 1 kudos
- 108 Views
- 2 replies
- 0 kudos
Power BI dashboards using Direct Lake
Hi Everyone, We are experimenting with using Direct Lake for our Power BI dashboards and I wanted to check in with this group to see if people have tried or are currently using this to connect PBI to Databricks tables. If so what is your experience w...
- 108 Views
- 2 replies
- 0 kudos
- 0 kudos
Your Power Bi report needs to be in Fabric capacity workspace in order for you leverage Direct Lake mode functionality.So you can try the path of Databricks mirroring :Fabric mirrors Unity catalog metadata and creates OneLake shortcuts to the Delta f...
- 0 kudos
- 294 Views
- 3 replies
- 0 kudos
Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 294 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , glad the first answer helped. Your sliding window observation is sharp, but one correction to the mental model: delta.deletedFileRetentionDuration only drives physical deletion at the moment a VACUUM executes. Between your w...
- 0 kudos
- 147 Views
- 1 replies
- 0 kudos
Resolved! Databricks delta tables and Iceberg
Hi experts ,now databricks for support for Iceberg open table format conversion yet not fully though.Databricks calls this Delta Lake UniForm (Universal Format). When tentatively Databricks can support fully open table format Iceberg, Hudi etc.Is the...
- 147 Views
- 1 replies
- 0 kudos
- 0 kudos
Worth separating "reading Iceberg from Delta" from "Databricks fully adopting Iceberg as a first-class table format" - those are two different things and Databricks has been moving on both fronts. Today: UniForm (Universal Format) makes a Delta table...
- 0 kudos
- 110 Views
- 1 replies
- 0 kudos
transformWithState might be causing 'Module not found'
Hi,So i've been playing around with transformWithState over the last few weeks and have got the code to work exactly as desired.Wrote some integration tests that execute fine within the workspace in serverless and everything, the issue is however is ...
- 110 Views
- 1 replies
- 0 kudos
- 0 kudos
This smells like a client/server module-resolution mismatch that's specific to Databricks Connect + serverless, rather than a wheel packaging problem per se. A few things worth checking: 1. Where the StatefulProcessor class lives. Since it's defined ...
- 0 kudos
- 68 Views
- 0 replies
- 0 kudos
Databricks Certification Exam Suspended - Databricks Certified Data Engineer Associate
Dear Databricks Certification Support Team,My Databricks Certified Data Engineer Associate exam was suspended during the exam session.After the exam was suspended, I verified my system and found that my webcam was not working properly. I believe the ...
- 68 Views
- 0 replies
- 0 kudos
- 1043 Views
- 6 replies
- 2 kudos
Code on cluster runs idefinitely
Hello.Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:- Cluster starts, have running status- I run code, for example print("Hello")- Code runs indefinitely- I click interrupt, it st...
- 1043 Views
- 6 replies
- 2 kudos
- 2 kudos
It sounds more like a cluster/driver initialization or resource issue than a Python problem, especially because even SELECT 1 hangs.I’d check these first:Look at the Driver logs and Event log for errors around the first notebook execution. Databricks...
- 2 kudos
- 587 Views
- 8 replies
- 1 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 587 Views
- 8 replies
- 1 kudos
- 1 kudos
Hey, Check the IAMs for both connector and ADLS, your problem will be solved...I have come across the same issue and have cleared it by assigning a role through IAM. LMK, if you have solved it Thanks
- 1 kudos
- 93 Views
- 0 replies
- 1 kudos
Streaming Doesn't Mean Your Compute Needs to Run Forever
When I first started working with streaming in Databricks, I had a very simple picture in my head:Streaming = keep the cluster running and process data continuously.New data arrives → Spark sees it → process it → repeat forever.Technically, that can ...
- 93 Views
- 0 replies
- 1 kudos
- 262 Views
- 4 replies
- 0 kudos
Creating Databricks agent
Hi All,We are trying to create a Databricks agent and were looking for a step by step approach if there is any. We did go through the Author an agent and deploy it on Databricks Apps | Databricks on AWS but weren't very clear on creation of a new age...
- 262 Views
- 4 replies
- 0 kudos
- 0 kudos
If you're approaching this specifically as a migration validation use case, I'd separate the agent into two layers: the validation tools and the agent reasoning layer.For example, expose controlled tools for querying Hive and Databricks, then have th...
- 0 kudos
- 12403 Views
- 10 replies
- 1 kudos
Error when reading delta lake files with Auto Loader
Hi,When reading Delta Lake file (created by Auto Loader) with this code: df = ( spark.readStream .format('cloudFiles') .option("cloudFiles.format", "delta") .option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint") .load(bronz...
- 12403 Views
- 10 replies
- 1 kudos
- 1 kudos
The key point is using the correct format for Delta tables rather than treating them like raw files. The same idea of using the right tool for the job applies in other technical workflows too, such as a delta executor apk when handling Android relate...
- 1 kudos
- 444 Views
- 4 replies
- 2 kudos
Resolved! Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI"
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
- 444 Views
- 4 replies
- 2 kudos
- 2 kudos
Hello, a quick update on thisI reached out to the databricks support team and they confirm that Unity Catalog service credentials do not support "api://" URIs. this is a known product limitation. They asked me to file for a feature request with Datab...
- 2 kudos
- 171 Views
- 2 replies
- 0 kudos
how to enable column level lineage in databricks
I have bronze layer data which is directly referred in silver. there are one or two columns from bronze layer which are used to build silver column.is there any way I can have column-level lineage shown in databricks like table lineage@DATA @lineage
- 171 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @prathamesh1982 , Good news: there's nothing to enable. @balajij8 has it right, Unity Catalog captures column-level lineage automatically. I'll add some prerequisites and gotchas to check if the lineage doesn't appear. If your bronze and silver t...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
