- 41 Views
- 1 replies
- 0 kudos
How to change a field when instancing cluster defined as variable?
I define all clusters as variable in separate files, so I can re-use them. Then I am accessing them in jobs as: The issue is that I want to change just the custom_tags in the cluster when instancing it for a job, cause my tags are different for each ...
- 41 Views
- 1 replies
- 0 kudos
- 0 kudos
Yes, you can achieve this seamlessly, but not by overriding the custom_tags inside the cluster variable. Instead, you define your specific tags at the Job level, and Databricks automatically merges them with your cluster variable's tags.Because compl...
- 0 kudos
- 103 Views
- 3 replies
- 1 kudos
Best practice to log Autoloader UNKNOWN_FIELD_EXCEPTION
Hi, When schema evolution is detected, Auto Loader throws an UNKNOWN_FIELD_EXCEPTION, and the error message includes schema information along with other related details. However, when I log the full message, it is too long and contains information th...
- 103 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @bi_123, I would avoid parsing the full rendered UNKNOWN_FIELD_EXCEPTION message. Databricks explicitly notes in the error-handling documentation that the rendered and parameterised messages are not stable across releases, so any logic that depend...
- 1 kudos
- 54 Views
- 1 replies
- 0 kudos
ANALYZE TABLE ... COMPUTE STATISTICS FOR ALL COLUMNS silently does not update column stats on DBR 15
Hi Databricks Support,We need help confirming whether this is a known DBR 15.4 LTS bug or an unsupported/configuration-specific behavior.SummaryOn a Databricks Runtime 15.4.40 Photon job cluster, `ANALYZE TABLE <catalog>.<schema>.<table> COMPUTE STAT...
- 54 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sd1700092, From what I can verify, this looks more like a DBR 15.4 job-cluster issue than expected behaviour. The public ANALYZE TABLE documentation is clear that ANALYZE TABLE ... COMPUTE STATISTICS FOR ALL COLUMNS applies to both Databricks Run...
- 0 kudos
- 653 Views
- 3 replies
- 0 kudos
Databricks Access Issue with UC
Hi All ,We are facing issues while reading Storage account where stream data from data verse in Unity catalog through External table but not every time . It was running fine with hiveAn error occurred while calling o393.sql.: org.apache.spark.SparkEx...
- 653 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Jothia, Apologies for the very delayed response here. Appreciate this was raised in November 2025. I wanted to close the loop in case you were still expecting an answer. From what you described, this does not look like a straightforward Unity Cat...
- 0 kudos
- 120 Views
- 1 replies
- 5 kudos
My experience replacing a Postgres → Kafka → DMS → S3 pipeline with Lakeflow Connect
Sharing my hands-on experience with Lakeflow Connect for anyone evaluating it for database ingestion. I recently moved data from PostgreSQL on AWS RDS into Databricks, and it replaced a painful legacy pipeline. Keeping this simple and practical.What ...
- 120 Views
- 1 replies
- 5 kudos
- 5 kudos
Great article!
- 5 kudos
- 74 Views
- 1 replies
- 0 kudos
Can we do a shallow clone on top of a shallow clone!
Case 1 Can we do a shallow clone on top of a shallow clone? If I do so, What would be the result Case 2 when ever the table over writes with new data how does it work.Case 3 when a the table is dropped and loaded then how does this scenario effect ...
- 74 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @damodhargandha, A good way to think about a shallow clone is that it copies the table’s metadata but still points to the source table's data files rather than copying them. Databricks explains that behaviour in the Clone a table " section of the ...
- 0 kudos
- 11630 Views
- 9 replies
- 1 kudos
Error when reading delta lake files with Auto Loader
Hi,When reading Delta Lake file (created by Auto Loader) with this code: df = ( spark.readStream .format('cloudFiles') .option("cloudFiles.format", "delta") .option("cloudFiles.schemaLocation", f"{silver_path}/_checkpoint") .load(bronz...
- 11630 Views
- 9 replies
- 1 kudos
- 1 kudos
I've been having similar issues with reading Delta Lake files and I think the solution lies in adjusting the format option. When working with Delta tables, it's essential to use the correct format to avoid compatibility issues. This reminds me of the...
- 1 kudos
- 96 Views
- 1 replies
- 0 kudos
databricks-connect library for python and pandas 3
Hello,databricks-connect is pinning pandas during the installation. Since we're moving towards pandas 3 can you please add the support for the newest version as well?
- 96 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Albertino , how are you doing today?Thanks for calling this out. as per my understanding, at the moment, the pandas pin is there intentionally. Databricks Connect release notes say supported pandas versions are currently limited to 1.0.5<=pandas<...
- 0 kudos
- 85 Views
- 1 replies
- 1 kudos
Databricks apps
I have multiple Databricks Apps running, but their usage is not fixed or predictable. Some apps are used only occasionally, while others may remain idle for long periods.Since Databricks Apps need to stay up and continue consuming resources even when...
- 85 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @anmolhhns, I couldn't find any public documentation showing that Databricks Apps supports automatic idle shutdown or usage-based scale-to-zero for the app runtime itself. The current documented lifecycle is that an app can be Running, Stopped, De...
- 1 kudos
- 131 Views
- 1 replies
- 0 kudos
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate First
From RAG Demo to Production on Databricks: 7 Things Teams Should Validate FirstBy Naveen AyallaMany teams can build a RAG demo quickly.Upload documents, create embeddings, connect a model, ask a question, and show an answer.But production is differen...
- 131 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for reading. I’m especially interested in hearing from people who have worked on real RAG or GenAI workflows.Which one has been the biggest challenge for your team?1. Choosing the right source data2. Access control and governance3. Improving r...
- 0 kudos
- 172 Views
- 3 replies
- 0 kudos
[Lakeflow Spark Declarative Pipelines] - Compatibility Mode not working
I’m working with an SDP pipeline that creates a streaming table using the dlt.create_streaming_table decorator. My goal is to expose this table through an external location so that a client can read it from Snowflake.I attempted to configure this dir...
- 172 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Jotaefe1991 ,The overlap error you are hitting is a Unity Catalog storage collision, not a DLT limitation.Here is exactly what is happening and how to fix it:The path you provided for "delta.universalFormat.compatibility.location" (abfss://.../br...
- 0 kudos
- 163 Views
- 3 replies
- 3 kudos
Resolved! json file existing in volume but not showing in UI
I have some json files existing in a specific volume when I try to search for them they don't appear but when I query the the volume using python I am able to get them and read their content.Any help ?
- 163 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @amirabedhiafi ,Catalog Explorer search won't return these files. This is likely because raw files in Volumes can change rapidly and aren't tracked in the system tables in the same way structured data is.Instead, I would suggest using a Genie Spac...
- 3 kudos
- 221 Views
- 2 replies
- 2 kudos
Resolved! Recurring Historical Data Modeling Patterns
After reviewing a surprising number of Databricks discussions around SCD2, CDC, historical reporting and temporal joins, I noticed that most historical data modeling challenges seem to fall into a small set of recurring patterns:Historical BackfillLa...
- 221 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello!I would add a few more historical modeling patterns that often appear separately, even though they overlap with SCD2, CDC, or temporal joins.One important case is bi-temporal modeling, where you need to separate business effective time from sys...
- 2 kudos
- 3917 Views
- 1 replies
- 2 kudos
re: Welcoming Bladebridge to Databricks!
Hi @Sujitha and Databricks team,Congrats on the acquisition of Bladebridge. We used this tool a couple years back to migrate an important ETL process from Informatica. I'm glad to see its part of the Data Intelligence Platform and have already take...
- 3917 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi,Thank you for sharing this feedback, and great to hear that you have already used BladeBridge successfully in a previous Informatica migration.I agree that a dedicated BladeBridge forum or community section would be useful, especially now that mor...
- 2 kudos
- 658 Views
- 1 replies
- 1 kudos
New remote (dbfs) caching python library
I had some problems getting much speedup at all from spark or DB disk cache, which I think is essential when developing PySpark code iteratively in notebooks. So I developed a handy caching-library for this which has recently been open sourced, see h...
- 658 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for sharing this. It looks useful, especially for iterative notebook development where the expensive part is not just reading source files but recomputing a complex intermediate DataFrame after several joins or transformations.I can see the va...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 572 | |
| 349 | |
| 287 |
