- 6460 Views
- 6 replies
- 8 kudos
[UNRESOLVED_ROUTINE] Cannot resolve function `date_format`
Hi all,We are getting the following error log in a Workflow:AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function `date_format` on search path [`system`.`builtin`, `system`.`session`]. SQLSTATE: 42883The Workflow consists in different noteb...
- 6460 Views
- 6 replies
- 8 kudos
- 8 kudos
Having a similar issue when moving from DBR 13 to DBR 16.4 in a multi-task Job. The first task works ok but the 2nd with identical code but different params fails. For me it fails with Cannot resolve function `and` (I assume from the .between(from_da...
- 8 kudos
- 180 Views
- 5 replies
- 0 kudos
Lakeflow Connect - Community Custom Connector - How to troubleshoot source ingestion logic "live"
Hi there, I'm a bit confused about how I should be performing "live" troubleshooting with Lakeflow Connect with a Community Custom Connector I'm working on. Below is a simple example where I attempt to update the ingestion type from "cdc_with_deletes...
- 180 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @ChristianRRL ,Your compute cluster is caching the older version of your Python package in memory. Standard job runs or dry runs don't always force a rebuild or reload of workspace files.I hope below steps will helps you,Trigger a Full Refresh: In...
- 0 kudos
- 401 Views
- 4 replies
- 4 kudos
Genie Ontology
One question that came to mind regarding Genie Ontology is its level of automated curation. From what I've seen, Databricks appears to be taking a different approach from many vendors, where ontologies are often manually designed.Are there meaningful...
- 401 Views
- 4 replies
- 4 kudos
- 4 kudos
Today, Genie Ontology provides per-answer citations, but not yet a full reviewable changelog or diff of newly learned context. For regulated environments, that means customers can see what informed an answer, but not yet review every underlying ontol...
- 4 kudos
- 214 Views
- 2 replies
- 0 kudos
Real time pipelines
How do you design high-volume real-time pipelines that need to stay reliable under load spikes?
- 214 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, This is a broad question without much detail. The first thing I'd want to understand is what is the definition of realtime that is required. People say realtime but often they can mean a variety of things: What is the latency requirement and wher...
- 0 kudos
- 302 Views
- 1 replies
- 1 kudos
Trunked-based development model for Databricks question
Hi there,My name is William Do (a senior engineer at Hub24, Australia). I am writing to seek for your advice on a Trunk-based development model that we are hoping to implement for our Databricks project.Problem statementTo give you some context, I am...
- 302 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi William,The release train in your GitFlow workflow model is what stops you iterating, and long-running feature branches are what jam up a naive trunk-based model.Where I would gently push back is on the proposed hybrid. Reintroducing a develop bra...
- 1 kudos
- 435 Views
- 3 replies
- 4 kudos
AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=`
Hi all,We are experiencing an AnalysisException when migrating our Databricks cluster runtime from 13.3 LTS to 16.4 LTS (Scala 2.13), and would appreciate your help in identifying the root cause.**Error:**```AnalysisException: [UNRESOLVED_ROUTINE] Ca...
- 435 Views
- 3 replies
- 4 kudos
- 4 kudos
Having a similar issue when moving from DBR 13 to DBR 16.4 in a multi-task Job. The first task works ok but the 2nd with identical code but different params fails. For me it fails with Cannot resolve function `and` (I assume from the .between(from_da...
- 4 kudos
- 77 Views
- 1 replies
- 1 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 77 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @adhi_databricks !This is a very specific and subtle architectural edge case when migrating traditional Scala JAR workloads to Databricks Serverless Compute.The core issue comes down to how Databricks Serverless Jobs execute code via Spark Connect...
- 1 kudos
- 4128 Views
- 10 replies
- 3 kudos
Resolved! Unable to create secret scope -"Fetch request failed due expired user session"
Hi everyone,I’m trying to create an Azure Key Vault-backed secret scope in a Databricks Premium workspace, but I keep getting this error: Fetch request failed due expired user sessionSetup details:Databricks workspace: PremiumAzure Key Vault: Owner p...
- 4128 Views
- 10 replies
- 3 kudos
- 3 kudos
Hi @lofidev I resolved by requesting a policy exemption window to create scopes. The issue was due to a restrictive policy. Precisely what @lingareddy_Alva summarized earlier on this thread "When Databricks tries to programmatically grant its SP Get/...
- 3 kudos
- 2368 Views
- 3 replies
- 1 kudos
Databricks Vs Fabric use case
Hi Team,We've noticed that for some use cases, customers are proposing a architecture with A) Fabric in the Gold layer and reporting in Azure Power BI, while using Databricks for the Bronze and Silver layers. However, we can also have the B) Gold lay...
- 2368 Views
- 3 replies
- 1 kudos
- 1 kudos
When looking at a data fabric vs data lake strategy, the data lake approach shines when you need a high-performance landing zone for vast amounts of structured, semi-structured, and unstructured data. It gives data science and engineering teams tota...
- 1 kudos
- 1475 Views
- 4 replies
- 1 kudos
Resolved! Design Oracle Fusion SCM to Azure Databricks
Hello Techie,I am planning to migrate All module of Oracle fusion scm data to Azure Databricks.Do we have only option of BICC (Business Intelligence Cloud Connector), OR any other option avaialble.Can anyone please help me with reference architecture...
- 1475 Views
- 4 replies
- 1 kudos
- 1 kudos
BICC is the standard choice for bulk Oracle Fusion SCM extraction, but REST APIs and OIC are also good options depending on your use case. Your architecture and data freshness requirements will determine the best approach.
- 1 kudos
- 165 Views
- 1 replies
- 1 kudos
R plots are not rendering properly again
It looks like we have another workspace-side notebook UI regression for R plots (something similar happened last May).This time, R plots are displaying but have several very strange symptoms related to scaling:1. Larger figures (that are too big to b...
- 165 Views
- 1 replies
- 1 kudos
- 1 kudos
My Notebook, which previously worked fine, is now scaling all plots to `repr.plot.width` by `repr.plot.width`.
- 1 kudos
- 12664 Views
- 11 replies
- 9 kudos
Resolved! What are powerfull data quality tools/libraries to build data quality framework in Databricks ?
Dear Community Experts,I need your expert advice and suggestions on development of data quality framework. What are powerfull data quality tools or libraries are good to go for development of data quality framework in Databricks ? Please guide team.R...
- 12664 Views
- 11 replies
- 9 kudos
- 9 kudos
I have used Databricks labs - DQX for a failrly complex data quality implementation and it handlled the mix of built in validations and custom business rules well. One thing we learned is that performance depended much more on the underlying data lay...
- 9 kudos
- 92 Views
- 0 replies
- 0 kudos
Lakeflow Connect SQL Server gateway – intermittent Entra ID token auth failures (18456, state 132)
Hi all,We run a Lakeflow Connect SQL Server ingestion pipeline with the gateway in continuous mode, reading from an Azure SQL Database using an Entra ID service principal (OAuth2 client credentials). The setup works overall, but we see a constant low...
- 92 Views
- 0 replies
- 0 kudos
- 314 Views
- 4 replies
- 2 kudos
Auto-Termination Did Not Trigger on Production Cluster Despite 20-Minute Inactivity
Hello everyone,I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.The cluster was configured with "Terminate after 20 minutes of inactivity", but it never termin...
- 314 Views
- 4 replies
- 2 kudos
- 2 kudos
@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.(1) Driver logs a...
- 2 kudos
- 96 Views
- 1 replies
- 0 kudos
Installing App from Marketplace
Hi All,I'm using the free edition and I'm trying to install a free app from the Marketplace. After the installation, it's asking me to provide secrets.Could you please guide me on where I can obtain these secrets? Do I need to get them from the app o...
- 96 Views
- 1 replies
- 0 kudos
- 0 kudos
Good question, and the answer is usually simpler than it looks: those secrets are almost always YOUR OWN credentials for whatever third-party service the app talks to — not something the developer sends you.Here's the model: Marketplace apps aren't a...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
