cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Steveo
by New Contributor II
  • 756 Views
  • 3 replies
  • 0 kudos

In Databricks, when saving a query, the changes do no persist. Any advice?

When returning to the query, the changes have reverted.Is this because the query is open by another user?How can this be resolved? #databricks

  • 756 Views
  • 3 replies
  • 0 kudos
Latest Reply
Steveo
New Contributor II
  • 0 kudos

We ARE using the save feature. But we are still loosing changes. Is this because we have multiple Databricks tabs open in separate Chrome browsers? Across 2 users.

  • 0 kudos
2 More Replies
gazzyjuruj
by Contributor II
  • 10249 Views
  • 14 replies
  • 11 kudos

Resolved! Failed to start cluster

Hi, I ran the cluster more than 5-6 times with it failing to start since this past morning (about 11-12 hours now) since i'm facing this problem.Attaching screenshot below and also typing in case someone comes from the web to this thread in future.Pr...

IMG_2152 IMG_2151
  • 10249 Views
  • 14 replies
  • 11 kudos
Latest Reply
worthpicker
New Contributor II
  • 11 kudos

The cluster will be able to start and the nodes will automatically obtain the updated cluster configuration data.

  • 11 kudos
13 More Replies
Anonymous
by Not applicable
  • 854 Views
  • 0 replies
  • 1 kudos

7 Steps For Mobile Banking App Development Banking industry experts have predicted that as soon as the pandemic is over, the banking institutions will...

7 Steps For Mobile Banking App DevelopmentBanking industry experts have predicted that as soon as the pandemic is over, the banking institutions will have to reform hooeyapps their operations by inclining towards digital banking. In the recent past, ...

  • 854 Views
  • 0 replies
  • 1 kudos
Siegel_Af_
by New Contributor
  • 339 Views
  • 0 replies
  • 0 kudos

playcasinosnj.com

One of the best platforms where you can find games like online slots is https://playcasinosnj.com/. It is in my opinion the safest site to find the games that suit you. I also had the chance to try out the different games and they all worked for me. ...

  • 339 Views
  • 0 replies
  • 0 kudos
labromb
by Contributor
  • 1407 Views
  • 0 replies
  • 0 kudos

Converting a text widget to a list

Hi, Been working on some parallel notebook code, which I have ported to python from the example on the DB website and added some exception handling and that works fine. What I would like to do is paramterise the input but am not succeeding as the fun...

  • 1407 Views
  • 0 replies
  • 0 kudos
zach
by New Contributor III
  • 1879 Views
  • 5 replies
  • 1 kudos

Does Databricks have a google cloud Big Query equivalent of --dry_run to estimate costs before executing?

Databricks uses DBU's as a costing unit whether based onto of AWS/Azure/GCP and I want to know if Databricks has a google cloud Big Query equivalent of --dry_run for estimating costs? https://cloud.google.com/bigquery/docs/estimate-costs

  • 1879 Views
  • 5 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @zach welshman​ ​, We haven’t heard from you on the last response from @Werner Stinckens​, and I was checking back to see if you have a resolution yet. If you have any solution, please share it with the community as it can be helpful to others. Ot...

  • 1 kudos
4 More Replies
alonisser
by Contributor
  • 6308 Views
  • 14 replies
  • 7 kudos

Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...

  • 6308 Views
  • 14 replies
  • 7 kudos
Latest Reply
alonisser
Contributor
  • 7 kudos

Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?

  • 7 kudos
13 More Replies
Taha_Hussain
by Valued Contributor II
  • 895 Views
  • 0 replies
  • 4 kudos

Databricks Office Hours Register for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for Ju...

Databricks Office HoursRegister for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for June 22nd from 8:00 am - 9:00am PT.This is your opportunity to connect directly with our experts to ask any...

  • 895 Views
  • 0 replies
  • 4 kudos
sreedata
by New Contributor III
  • 2138 Views
  • 3 replies
  • 5 kudos

Resolved! Databricks -->Workflows-->Job Runs

In Databricks -->Workflows-->Job Runs we have a column "Run As".From where does this value come. We are getting a user id here but need to change it to a generic account. Any help would be appreciated. Thanks

  • 2138 Views
  • 3 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

I appreciate the information and advice you have shared.DNAHRBlock.com

  • 5 kudos
2 More Replies
wyzer
by Contributor II
  • 3080 Views
  • 4 replies
  • 2 kudos

Resolved! Unable to delete a DBFS folder

Hello everyone,I've created by error a DBFS folder named : ${env]But when I run this command :dbutils.fs.rm("/mnt/${env]")It returns me this error : java.net.URISyntaxException: Illegal character in path at index 12: /mnt/$%7Benv]How can I do please ...

  • 3080 Views
  • 4 replies
  • 2 kudos
Latest Reply
User16764241763
Honored Contributor
  • 2 kudos

Hello @Salah K.​ Can you try below?%sh rm -r /dbfs/mnt/$\{env\]

  • 2 kudos
3 More Replies
nadia
by New Contributor II
  • 1124 Views
  • 1 replies
  • 0 kudos

Resolved! Connection Databricks Postgresql

I use Databricks and I try to connect to posgresql via the following code"jdbcHostname = "xxxxxxx"jdbcDatabase = "xxxxxxxxxxxx"jdbcPort = "5432"username = "xxxxxxx"password = "xxxxxxxx"jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jd...

  • 1124 Views
  • 1 replies
  • 0 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 0 kudos

hi @Boumaza nadia​ Please check the Ganglia metrics for the cluster. This could be a scalability issue where cluster is overloading. This can happen due to a large partition not fitting into the given executor's memory. To fix this we recommend bump...

  • 0 kudos
shubhamb
by New Contributor III
  • 3016 Views
  • 4 replies
  • 6 kudos

Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks

My function in func.pydef lower_events(df): return df.withColumn("event",f.lower(f.col("event")))My main notebook import pyspark.sql.functions as f from pyspark.sql.functions import udf, col, lower import sys   sys.path.append("..") from folder.func...

  • 3016 Views
  • 4 replies
  • 6 kudos
Latest Reply
shubhamb
New Contributor III
  • 6 kudos

@Kaniz Fatma​  https://community.databricks.com/s/question/0D58Y00008ouo6xSAA/how-to-fetch-environmental-variables-saved-in-one-notebook-into-another-notebook-in-databricks-repos-and-notebooksCan you please look into this

  • 6 kudos
3 More Replies
mortenhaga
by Contributor
  • 1877 Views
  • 2 replies
  • 2 kudos

Resolved! Importing python function with spark.read.jdbc in to Repos

Hi all!Before we used Databricks Repos we used the run magic to run various utility python functions from one notebook inside other notebooks, fex like reading from a jdbc connections. We now plan to switch to repos to utilize the fantastic CI/CD pos...

image.png image
  • 1877 Views
  • 2 replies
  • 2 kudos
Latest Reply
mortenhaga
Contributor
  • 2 kudos

Thats...odd. I was sure I had tried that, but now it works somehow. I guess it has to be that now I did it with double quotation marks. Thanks anyway! Works like a charm.

  • 2 kudos
1 More Replies
Marra
by New Contributor III
  • 3954 Views
  • 8 replies
  • 2 kudos

Read temporary views in SQL Analytics

I'm having issues trying to read temporary views in SQL Analytics module. Ive managed to create temporary views based on a query but I don't know how to read from them? Just using the name of the view returns "Table or view not found".

  • 3954 Views
  • 8 replies
  • 2 kudos
Latest Reply
Marra
New Contributor III
  • 2 kudos

No, I'm actually having issues reading from the view in the same session that created it. Using the same view name I get a table or view not found.

  • 2 kudos
7 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels