- 40 Views
- 2 replies
- 1 kudos
INVALID_PARAMETER_VALUE.LOCATION_OVERLAP after catalog deleted
I've got an ADLS Gen 2 container that has cloud files that I want to ingest with AutoLoader. The files are saved into the root of the container.I created an external location for that container to be able to access it through managed credentials. How...
- 40 Views
- 2 replies
- 1 kudos
- 1 kudos
I have not found a supported way to make Unity Catalog release that path early, but you can get Auto Loader running again today without it. The reservation belongs to the managed table your access test created. Dropping it starts a recovery window ra...
- 1 kudos
- 486 Views
- 5 replies
- 2 kudos
Resolved! Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI"
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
- 486 Views
- 5 replies
- 2 kudos
- 2 kudos
Hello, a quick update on thisI reached out to the databricks support team and they confirm that Unity Catalog service credentials do not support "api://" URIs. this is a known product limitation. They asked me to file for a feature request with Datab...
- 2 kudos
- 36 Views
- 2 replies
- 0 kudos
How to show full query results in a SQL alert's notification?
I have a SQL alert; the code behind it looks like this:SELECT COLUMN_1, COLUMN_2, COUNT(*) AS TOTAL, 1 AS FORCE_TRIGGERFROM MY_TABLEGROUP BY COLUMN_1, COLUMN_2The idea is that a scheduled job updates my_table and then runs the alert. The alert trigge...
- 36 Views
- 2 replies
- 0 kudos
- 0 kudos
You are getting one row because of the MIN in the alert condition rather than anything in the notification template. Setting an aggregation rewrites the query before it runs. From Alert query patterns: "The alert wraps the original query text in a co...
- 0 kudos
- 105 Views
- 1 replies
- 1 kudos
An AI agent that validates and PRs your Unity Catalog access requests, and knows when to say no
Most access requests still go through a person hand-writing Terraform off a Slack message and hoping the scope matches what was actually asked for.I spent the weekend building a pipeline that closes that gap. A Databricks Unity Catalog access request...
- 105 Views
- 1 replies
- 1 kudos
- 1 kudos
Honestly, the “make it repeatable” point is huge. The tool matters less than having a workflow you can rerun without manually fixing the same mess every time.
- 1 kudos
- 36 Views
- 0 replies
- 1 kudos
cross-region DR in Azure Databricks (24h RPO/RTO)
Hi,We're designing a DR strategy for an Azure Databricks platform and would appreciate guidance on current best practices for achieving approximately 24-hour RPO and RTO across Azure regions.Our platform includes Unity Catalog, DAB, Jobs, Notebooks, ...
- 36 Views
- 0 replies
- 1 kudos
- 5155 Views
- 6 replies
- 1 kudos
Resolved! Best practices for the meta data driven ETL framework
I am designing a metadata‑driven ETL framework to migrate approximately 500 tables from Db2 to PostgreSQL.After reviewing multiple design patterns and blog posts, I am uncertain about the recommended approach for storing ETL metadata such as source s...
- 5155 Views
- 6 replies
- 1 kudos
- 1 kudos
@nayan_wylde : I can absolutely relate and we also use a metadata database for our control tables. But how do you handle the content of the tables? Until now we just wrote insert scripts in a similar way that flyway uses migrations. The problem ist t...
- 1 kudos
- 312 Views
- 7 replies
- 2 kudos
Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 312 Views
- 7 replies
- 2 kudos
- 2 kudos
I am curious - both @adnan_alvee and @ShamenParis has slightly different approach and both makes sense. Perhaps an open question - is the former more idiomatic even though both could work in this case?
- 2 kudos
- 237 Views
- 1 replies
- 0 kudos
Transaction log integrity issue
Delta transaction log for one of our tables which is being written to by a Kafka Connect IcebergSinkConnector via the Unity Catalog Iceberg REST endpoint, is currently corrupted and is failing when trying to read with the below exceptionERROR:com.dat...
- 237 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @analyticsnerd , Delta maintains a checksum file (.crc) for each committed version, along with the commit log (.json), recording the table’s file count, total size, and a distribution of file sizes (a histogram that buckets files by size). For ...
- 0 kudos
- 293 Views
- 2 replies
- 1 kudos
Cross-engine Iceberg V3 compatibility issue- Databricks DELETE deletion vectors not readable by ext
Hi Databricks Support Team,We are raising this ticket to report a cross-engine compatibility issue between Databricks and Teradata when performing DELETE operations on an Iceberg V3 table.---ENVIRONMENT DETAILS-------------------Databricks Runtime V...
- 293 Views
- 2 replies
- 1 kudos
- 1 kudos
This appears to be a cross-engine Iceberg V3 compatibility issue. If Databricks writes .bin deletion vectors while Teradata only supports Puffin, the key question is whether Puffin output can be enabled in DBR 17.3. I’d ask Databricks to confirm curr...
- 1 kudos
- 156 Views
- 2 replies
- 0 kudos
Power BI dashboards using Direct Lake
Hi Everyone, We are experimenting with using Direct Lake for our Power BI dashboards and I wanted to check in with this group to see if people have tried or are currently using this to connect PBI to Databricks tables. If so what is your experience w...
- 156 Views
- 2 replies
- 0 kudos
- 0 kudos
Your Power Bi report needs to be in Fabric capacity workspace in order for you leverage Direct Lake mode functionality.So you can try the path of Databricks mirroring :Fabric mirrors Unity catalog metadata and creates OneLake shortcuts to the Delta f...
- 0 kudos
- 335 Views
- 3 replies
- 0 kudos
Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 335 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , glad the first answer helped. Your sliding window observation is sharp, but one correction to the mental model: delta.deletedFileRetentionDuration only drives physical deletion at the moment a VACUUM executes. Between your w...
- 0 kudos
- 193 Views
- 1 replies
- 0 kudos
Resolved! Databricks delta tables and Iceberg
Hi experts ,now databricks for support for Iceberg open table format conversion yet not fully though.Databricks calls this Delta Lake UniForm (Universal Format). When tentatively Databricks can support fully open table format Iceberg, Hudi etc.Is the...
- 193 Views
- 1 replies
- 0 kudos
- 0 kudos
Worth separating "reading Iceberg from Delta" from "Databricks fully adopting Iceberg as a first-class table format" - those are two different things and Databricks has been moving on both fronts. Today: UniForm (Universal Format) makes a Delta table...
- 0 kudos
- 123 Views
- 1 replies
- 0 kudos
transformWithState might be causing 'Module not found'
Hi,So i've been playing around with transformWithState over the last few weeks and have got the code to work exactly as desired.Wrote some integration tests that execute fine within the workspace in serverless and everything, the issue is however is ...
- 123 Views
- 1 replies
- 0 kudos
- 0 kudos
This smells like a client/server module-resolution mismatch that's specific to Databricks Connect + serverless, rather than a wheel packaging problem per se. A few things worth checking: 1. Where the StatefulProcessor class lives. Since it's defined ...
- 0 kudos
- 96 Views
- 0 replies
- 0 kudos
Databricks Certification Exam Suspended - Databricks Certified Data Engineer Associate
Dear Databricks Certification Support Team,My Databricks Certified Data Engineer Associate exam was suspended during the exam session.After the exam was suspended, I verified my system and found that my webcam was not working properly. I believe the ...
- 96 Views
- 0 replies
- 0 kudos
- 1059 Views
- 6 replies
- 2 kudos
Code on cluster runs idefinitely
Hello.Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:- Cluster starts, have running status- I run code, for example print("Hello")- Code runs indefinitely- I click interrupt, it st...
- 1059 Views
- 6 replies
- 2 kudos
- 2 kudos
It sounds more like a cluster/driver initialization or resource issue than a Python problem, especially because even SELECT 1 hangs.I’d check these first:Look at the Driver logs and Event log for errors around the first notebook execution. Databricks...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
