cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Dicer
by • Valued Contributor
  • 3512 Views
  • 5 replies
  • 5 kudos

Resolved! Azure Databricks: AnalysisException: Database 'bf' not found

I wanted to save my delta tables in my Databricks database. When I saveAsTable, there is an error message Azure Databricks: AnalysisException: Database 'bf' not found​Ye, There is no database named "bf" in my database.Here is my full code:import os i...

Data Engineering
Reply
  • 3512 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
Dicer
Valued Contributor
  • 5 kudos

Some data can be saved as delta tables while some cannot.

  • 5 kudos
4 More Replies
mani238
by • New Contributor III
  • 14894 Views
  • 2 replies
  • 2 kudos

Resolved! Run result unavailable: job failed with error message Library installation failed for library due to user error for jar: "dbfs:/my-jar.jar"

Run result unavailable: job failed with error message Library installation failed for library due to user error for jar: "dbfs:/my-jar.jar" . Error messages: Library installation attempted on the driver node of cluster 0510-013936-3cc6d9kw and failed...

Data Engineering
Reply
  • 14894 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hey there @manivannan p​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from yo...

  • 2 kudos
1 More Replies
Dataminion
by • New Contributor
  • 3577 Views
  • 1 replies
  • 0 kudos

No module named dlt

Cluster version 10.5 on AWS.Trying to import dlt gave error above.​Am I missing something?​

Data Engineering
Reply
  • 3577 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hey there @Cathy Low​ Does @Kaniz Fatma​ 's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? Else please let us know if you need more help. We'd love to hear from y...

  • 0 kudos
SailajaB
by • Valued Contributor III
  • 4139 Views
  • 2 replies
  • 6 kudos

Resolved! How to pass a variable which holds a value to child notebook using run command

Hello,We have 3 notebooks as below. And trying to send a variable where we pass the value to it via job to child notebook with %run commandNotebooks:notebook_par,notebook_child1 and notebook_child2.In parent note book we are calling notebook_child1 u...

Data Engineering
Reply
  • 4139 Views
  • 2 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hey there @Sailaja B​ How's it going?Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Chee...

  • 6 kudos
1 More Replies
prasadvaze
by • Valued Contributor II
  • 38513 Views
  • 10 replies
  • 8 kudos

When to use delta lake versus relational database as a source for BI reporting?

Assume all of your data exists in delta tables and also in SQL server so you have a choice to report from either. Can someone share thoughts on "In what scenario you would not want report created from delta table and instead use the traditional rel...

Data Engineering
Reply
  • 38513 Views
  • 10 replies
  • 8 kudos
View Replies
Latest Reply
PCJ
New Contributor II
  • 8 kudos

Hi @Kaniz Fatma​  - I would like a follow-up on @prasad vaze​ question regarding unsupported referential integrity. How does one work around that, using best practices as Databricks sees it?

  • 8 kudos
9 More Replies
Steveo
by • New Contributor II
  • 1160 Views
  • 1 replies
  • 0 kudos

In Databricks, when saving a query, the changes do no persist. Any advice?

When returning to the query, the changes have reverted.Is this because the query is open by another user?How can this be resolved? #databricks

Data Engineering
Reply
  • 1160 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Steveo
New Contributor II
  • 0 kudos

We ARE using the save feature. But we are still loosing changes. Is this because we have multiple Databricks tabs open in separate Chrome browsers? Across 2 users.

  • 0 kudos
gazzyjuruj
by • Contributor II
  • 17700 Views
  • 9 replies
  • 8 kudos

Resolved! Failed to start cluster

Hi, I ran the cluster more than 5-6 times with it failing to start since this past morning (about 11-12 hours now) since i'm facing this problem.Attaching screenshot below and also typing in case someone comes from the web to this thread in future.Pr...

IMG_2152 IMG_2151
Data Engineering
Reply
  • 17700 Views
  • 9 replies
  • 8 kudos
View Replies
Latest Reply
worthpicker
New Contributor II
  • 8 kudos

The cluster will be able to start and the nodes will automatically obtain the updated cluster configuration data.

  • 8 kudos
8 More Replies
Anonymous
by • Not applicable
  • 1496 Views
  • 0 replies
  • 1 kudos

7 Steps For Mobile Banking App Development Banking industry experts have predicted that as soon as the pandemic is over, the banking institutions will...

7 Steps For Mobile Banking App DevelopmentBanking industry experts have predicted that as soon as the pandemic is over, the banking institutions will have to reform hooeyapps their operations by inclining towards digital banking. In the recent past, ...

Data Engineering
Reply
  • 1496 Views
  • 0 replies
  • 1 kudos
labromb
by • Contributor
  • 2063 Views
  • 0 replies
  • 0 kudos

Converting a text widget to a list

Hi, Been working on some parallel notebook code, which I have ported to python from the example on the DB website and added some exception handling and that works fine. What I would like to do is paramterise the input but am not succeeding as the fun...

Data Engineering
Reply
  • 2063 Views
  • 0 replies
  • 0 kudos
anmol_deep
by • New Contributor III
  • 3384 Views
  • 2 replies
  • 2 kudos

How to restore DatabricksRoot(FileStore) data after Databricks Workspace is decommissioned?

My Azure Databricks workspace was decommissioned. I forgot to copy files stored in the DatabricksRoot storage (dbfs:/FileStore/...).Can the workspace be recommissioned/restored? Is there any way to get my data back?Also, is there any difference betwe...

Data Engineering
Reply
  • 3384 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
User16764241763
Honored Contributor
  • 2 kudos

Hello @Anmol Deep​  Please submit a support request ASAP, so we can restore the deleted workspace. You can recover artifacts from the workspace.

  • 2 kudos
1 More Replies
zach
by • New Contributor III
  • 2917 Views
  • 4 replies
  • 1 kudos

Does Databricks have a google cloud Big Query equivalent of --dry_run to estimate costs before executing?

Databricks uses DBU's as a costing unit whether based onto of AWS/Azure/GCP and I want to know if Databricks has a google cloud Big Query equivalent of --dry_run for estimating costs? https://cloud.google.com/bigquery/docs/estimate-costs

Data Engineering
Reply
  • 2917 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

Not that I know of.Google uses number of bytes read to determine the cost.Databricks uses DBU. The number of DBU's spent is not only dependent on the amount of bytes read (the more you read, the longer the program will run probably), but also the typ...

  • 1 kudos
3 More Replies
BeginnerBob
by • New Contributor III
  • 2518 Views
  • 3 replies
  • 1 kudos

Loading Dimensions including SCDType2

I have a customer dimension and for every incremental load I am applying type2 or type1 to the dimension.This dimension is based off a silver table in my delta lake where I am applying a merge statement.What happens if I need to go back and track ad...

Data Engineering
Reply
  • 2518 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
BeginnerBob
New Contributor III
  • 1 kudos

Thanks werners,I was informed you could essentially recreate a type 2 dimensions from scratch, without reading the files 1 by 1, using the delta lake time shift. However, this doesn't seem to be the case and the only way to create this is to incremen...

  • 1 kudos
2 More Replies
alonisser
by • Contributor
  • 9027 Views
  • 8 replies
  • 6 kudos

Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...

Data Engineering
Reply
  • 9027 Views
  • 8 replies
  • 6 kudos
View Replies
Latest Reply
alonisser
Contributor
  • 6 kudos

Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?

  • 6 kudos
7 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels