cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Raj_DB
by • Contributor
  • 409 Views
  • 1 replies
  • 0 kudos

Power BI Refresh Fails with "Unable to Continue Fetch After Reconnect" from Azure Databricks

Hi everyone,I'm refreshing a report with dataset (~10 million rows) from Azure Databricks in power Bi service. The refresh runs for about an hour and then fails with: Unable to continue fetch after reconnect. Retry limit exceeded. I am currently unab...

IMG_0222.jpeg
  • 409 Views
  • 1 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor II
  • 0 kudos

Hi Raj,Query execution on Databricks completed in under a second but the failure is occurring during the client-side result streaming phase between Power BI and Azure Databricks.When importing Power BI streams result chunks over HTTP connections. If ...

  • 0 kudos
maddan80
by • New Contributor III
  • 5130 Views
  • 9 replies
  • 4 kudos

Oracle Essbase connectivity

Team, I wanted to understand the best way of connecting to Oracle Essbase to ingest data into the delta lake

  • 5130 Views
  • 9 replies
  • 4 kudos
Latest Reply
cartergray70543
Contributor II
  • 4 kudos

It depends on your Essbase setup. If it supports REST APIs or ODBC/JDBC, those are usually the best options for loading data into a Delta Lake. Share your Essbase version and target platform (e.g., Microsoft Fabric or Databricks) for more specific gu...

  • 4 kudos
8 More Replies
Jaclaglez13
by • New Contributor III
  • 7244 Views
  • 6 replies
  • 8 kudos

[UNRESOLVED_ROUTINE] Cannot resolve function `date_format`

Hi all,We are getting the following error log in a Workflow:AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function `date_format` on search path [`system`.`builtin`, `system`.`session`]. SQLSTATE: 42883The Workflow consists in different noteb...

  • 7244 Views
  • 6 replies
  • 8 kudos
Latest Reply
VaDim
Contributor
  • 8 kudos

Having a similar issue when moving from DBR 13 to DBR 16.4 in a multi-task Job. The first task works ok but the 2nd with identical code but different params fails. For me it fails with Cannot resolve function `and` (I assume from the .between(from_da...

  • 8 kudos
5 More Replies
ali_lmalki
by • New Contributor II
  • 2650 Views
  • 4 replies
  • 7 kudos

Genie Ontology

One question that came to mind regarding Genie Ontology is its level of automated curation. From what I've seen, Databricks appears to be taking a different approach from many vendors, where ontologies are often manually designed.Are there meaningful...

  • 2650 Views
  • 4 replies
  • 7 kudos
Latest Reply
Lu_Wang_ENB_DBX
Databricks Employee
  • 7 kudos

Today, Genie Ontology provides per-answer citations, but not yet a full reviewable changelog or diff of newly learned context. For regulated environments, that means customers can see what informed an answer, but not yet review every underlying ontol...

  • 7 kudos
3 More Replies
theanhdo
by • New Contributor III
  • 706 Views
  • 1 replies
  • 1 kudos

Trunked-based development model for Databricks question

Hi there,My name is William Do (a senior engineer at Hub24, Australia). I am writing to seek for your advice on a Trunk-based development model that we are hoping to implement for our Databricks project.Problem statementTo give you some context, I am...

  • 706 Views
  • 1 replies
  • 1 kudos
Latest Reply
snicholson-arch
New Contributor II
  • 1 kudos

Hi William,The release train in your GitFlow workflow model is what stops you iterating, and long-running feature branches are what jam up a naive trunk-based model.Where I would gently push back is on the proposed hybrid. Reintroducing a develop bra...

  • 1 kudos
phanvy5403
by • New Contributor III
  • 913 Views
  • 3 replies
  • 5 kudos

AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve routine `=`

Hi all,We are experiencing an AnalysisException when migrating our Databricks cluster runtime from 13.3 LTS to 16.4 LTS (Scala 2.13), and would appreciate your help in identifying the root cause.**Error:**```AnalysisException: [UNRESOLVED_ROUTINE] Ca...

  • 913 Views
  • 3 replies
  • 5 kudos
Latest Reply
VaDim
Contributor
  • 5 kudos

Having a similar issue when moving from DBR 13 to DBR 16.4 in a multi-task Job. The first task works ok but the 2nd with identical code but different params fails. For me it fails with Cannot resolve function `and` (I assume from the .between(from_da...

  • 5 kudos
2 More Replies
AnandGNR
by • New Contributor III
  • 5903 Views
  • 10 replies
  • 3 kudos

Resolved! Unable to create secret scope -"Fetch request failed due expired user session"

Hi everyone,I’m trying to create an Azure Key Vault-backed secret scope in a Databricks Premium workspace, but I keep getting this error: Fetch request failed due expired user sessionSetup details:Databricks workspace: PremiumAzure Key Vault: Owner p...

  • 5903 Views
  • 10 replies
  • 3 kudos
Latest Reply
AnandGNR
New Contributor III
  • 3 kudos

Hi @lofidev I resolved by requesting a policy exemption window to create scopes. The issue was due to a restrictive policy. Precisely what @lingareddy_Alva summarized earlier on this thread "When Databricks tries to programmatically grant its SP Get/...

  • 3 kudos
9 More Replies
Phani1
by • Databricks MVP
  • 2830 Views
  • 3 replies
  • 1 kudos

Databricks Vs Fabric use case

Hi Team,We've noticed that for some use cases, customers are proposing a architecture with A) Fabric in the Gold layer and reporting in Azure Power BI, while using Databricks for the Bronze and Silver layers. However, we can also have the B) Gold lay...

  • 2830 Views
  • 3 replies
  • 1 kudos
Latest Reply
priyasinhakt
New Contributor II
  • 1 kudos

When looking at a data fabric vs data lake  strategy, the data lake approach shines when you need a high-performance landing zone for vast amounts of structured, semi-structured, and unstructured data. It gives data science and engineering teams tota...

  • 1 kudos
2 More Replies
Datalight
by • Contributor
  • 1994 Views
  • 4 replies
  • 1 kudos

Resolved! Design Oracle Fusion SCM to Azure Databricks

Hello Techie,I am planning to migrate All module of Oracle fusion scm data to Azure Databricks.Do we have only option of BICC (Business Intelligence Cloud Connector), OR any other option avaialble.Can anyone please help me with reference architecture...

  • 1994 Views
  • 4 replies
  • 1 kudos
Latest Reply
masonreed11
Contributor
  • 1 kudos

BICC is the standard choice for bulk Oracle Fusion SCM extraction, but REST APIs and OIC are also good options depending on your use case. Your architecture and data freshness requirements will determine the best approach.

  • 1 kudos
3 More Replies
lkahn
by • New Contributor II
  • 975 Views
  • 1 replies
  • 1 kudos

R plots are not rendering properly again

It looks like we have another workspace-side notebook UI regression for R plots (something similar happened last May).This time, R plots are displaying but have several very strange symptoms related to scaling:1. Larger figures (that are too big to b...

lkahn_0-1784762090805.png
  • 975 Views
  • 1 replies
  • 1 kudos
Latest Reply
aviast
New Contributor II
  • 1 kudos

My Notebook, which previously worked fine, is now scaling all plots to `repr.plot.width` by `repr.plot.width`.

  • 1 kudos
van45678
by • New Contributor III
  • 909 Views
  • 4 replies
  • 2 kudos

Auto-Termination Did Not Trigger on Production Cluster Despite 20-Minute Inactivity

Hello everyone,I recently encountered an issue with one of our production all-purpose clusters where the auto-termination feature did not work as expected.The cluster was configured with "Terminate after 20 minutes of inactivity", but it never termin...

van45678_0-1784894552797.png van45678_1-1784894588839.png
  • 909 Views
  • 4 replies
  • 2 kudos
Latest Reply
binlogreader
Contributor
  • 2 kudos

@van45678 -- @balajij8 's diagnosis matches what your Spark UI shows, so I will just add three things from the operational side, one for the postmortem and two for making sure the next occurrence costs you hours instead of five days.(1) Driver logs a...

  • 2 kudos
3 More Replies
GvReddy
by • New Contributor
  • 597 Views
  • 1 replies
  • 0 kudos

Resolved! Installing App from Marketplace

Hi All,I'm using the free edition and I'm trying to install a free app from the Marketplace. After the installation, it's asking me to provide secrets.Could you please guide me on where I can obtain these secrets? Do I need to get them from the app o...

  • 597 Views
  • 1 replies
  • 0 kudos
Latest Reply
ThomazNeto
Databricks Partner
  • 0 kudos

Good question, and the answer is usually simpler than it looks: those secrets are almost always YOUR OWN credentials for whatever third-party service the app talks to — not something the developer sends you.Here's the model: Marketplace apps aren't a...

  • 0 kudos
Dhruv-22
by • Contributor III
  • 880 Views
  • 3 replies
  • 1 kudos

(Design Decision) Why VOID in STRUCT is not castable to VARIANT?

I asked a question recently about VOID in STRUCT failing to cast to VARIANT, whereas a VOID column is castable to VARIANT. LinkEarlier I thought it might be an error since cast was possible somehow. But the community members replied that this is the ...

Dhruv22_1-1785672645237.png Dhruv22_2-1785672677715.png Dhruv22_3-1785672691891.png Dhruv22_4-1785672772815.png
  • 880 Views
  • 3 replies
  • 1 kudos
Latest Reply
ThomazNeto
Databricks Partner
  • 1 kudos

I ran some tests.**Answer — verified at source level and across six runs, Spark 3.5.2 → 4.2.0(evidence dossier attached):**There are two independent rejections here sharing one error class, which iswhat made this confusing:1. `CAST(struct AS VARIANT)...

  • 1 kudos
2 More Replies
yutmurakami
by • New Contributor II
  • 1577 Views
  • 5 replies
  • 1 kudos

Databricks Apps - "App Not Available" error with locationId parameter missing

Hello,I'm experiencing an issue with Databricks Apps where I cannot access my deployed app from a browser, even though the app appears to be running successfully.**Symptoms:**- Browser shows "App Not Available" error when accessing the app URL- Conso...

  • 1577 Views
  • 5 replies
  • 1 kudos
Latest Reply
ThomazNeto
Databricks Partner
  • 1 kudos

Hi @yutmurakami — I can now answer your question with data instead of hypotheses. I ran your exact scenario on my own Free Edition workspace (also AWS, different workspace than yours), twice.Test 1: existing app on my Free Edition workspace — Running...

  • 1 kudos
4 More Replies
ChristianRRL
by • Honored Contributor II
  • 2457 Views
  • 2 replies
  • 0 kudos

Databricks "Preferred" Approaches To Backfilling Single Column In Wide Tables

Hi there,I've tried thinking through this and googling as well, but I'm not sure if there's a better approach that I might be missing. We have *wide* tables with hundreds of columns, and on a day-to-day basis these tables are incrementally filled in ...

  • 2457 Views
  • 2 replies
  • 0 kudos
Latest Reply
gaupia
New Contributor II
  • 0 kudos

We hit exactly this, offline feature store on Redshift, ~200-column denormalized table fed by domain models. Adding a feature meant NULLs for history, and full recompute was too expensive, so we landed on: recompute the domain for the backfill range ...

  • 0 kudos
1 More Replies
Labels