cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

alonisser
by Contributor
  • 9151 Views
  • 8 replies
  • 6 kudos

Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...

  • 9151 Views
  • 8 replies
  • 6 kudos
Latest Reply
alonisser
Contributor
  • 6 kudos

Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?

  • 6 kudos
7 More Replies
Taha_Hussain
by Databricks Employee
  • 1119 Views
  • 0 replies
  • 4 kudos

Databricks Office Hours Register for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for Ju...

Databricks Office HoursRegister for Office Hours to participate in a live Q&A session with Databricks experts! Our next event is scheduled for June 22nd from 8:00 am - 9:00am PT.This is your opportunity to connect directly with our experts to ask any...

  • 1119 Views
  • 0 replies
  • 4 kudos
wyzer
by Contributor II
  • 4050 Views
  • 4 replies
  • 2 kudos

Resolved! Unable to delete a DBFS folder

Hello everyone,I've created by error a DBFS folder named : ${env]But when I run this command :dbutils.fs.rm("/mnt/${env]")It returns me this error : java.net.URISyntaxException: Illegal character in path at index 12: /mnt/$%7Benv]How can I do please ...

  • 4050 Views
  • 4 replies
  • 2 kudos
Latest Reply
User16764241763
Honored Contributor
  • 2 kudos

Hello @Salah K.​ Can you try below?%sh rm -r /dbfs/mnt/$\{env\]

  • 2 kudos
3 More Replies
nadia
by New Contributor II
  • 1741 Views
  • 1 replies
  • 0 kudos

Resolved! Connection Databricks Postgresql

I use Databricks and I try to connect to posgresql via the following code"jdbcHostname = "xxxxxxx"jdbcDatabase = "xxxxxxxxxxxx"jdbcPort = "5432"username = "xxxxxxx"password = "xxxxxxxx"jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jd...

  • 1741 Views
  • 1 replies
  • 0 kudos
Latest Reply
Prabakar
Databricks Employee
  • 0 kudos

hi @Boumaza nadia​ Please check the Ganglia metrics for the cluster. This could be a scalability issue where cluster is overloading. This can happen due to a large partition not fitting into the given executor's memory. To fix this we recommend bump...

  • 0 kudos
shubhamb
by New Contributor III
  • 3902 Views
  • 2 replies
  • 3 kudos

Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks

My function in func.pydef lower_events(df): return df.withColumn("event",f.lower(f.col("event")))My main notebook import pyspark.sql.functions as f from pyspark.sql.functions import udf, col, lower import sys   sys.path.append("..") from folder.func...

  • 3902 Views
  • 2 replies
  • 3 kudos
Latest Reply
shubhamb
New Contributor III
  • 3 kudos

@Kaniz Fatma​  https://community.databricks.com/s/question/0D58Y00008ouo6xSAA/how-to-fetch-environmental-variables-saved-in-one-notebook-into-another-notebook-in-databricks-repos-and-notebooksCan you please look into this

  • 3 kudos
1 More Replies
mortenhaga
by Contributor
  • 2796 Views
  • 2 replies
  • 2 kudos

Resolved! Importing python function with spark.read.jdbc in to Repos

Hi all!Before we used Databricks Repos we used the run magic to run various utility python functions from one notebook inside other notebooks, fex like reading from a jdbc connections. We now plan to switch to repos to utilize the fantastic CI/CD pos...

image.png image
  • 2796 Views
  • 2 replies
  • 2 kudos
Latest Reply
mortenhaga
Contributor
  • 2 kudos

Thats...odd. I was sure I had tried that, but now it works somehow. I guess it has to be that now I did it with double quotation marks. Thanks anyway! Works like a charm.

  • 2 kudos
1 More Replies
Marra
by New Contributor III
  • 6227 Views
  • 7 replies
  • 2 kudos

Read temporary views in SQL Analytics

I'm having issues trying to read temporary views in SQL Analytics module. Ive managed to create temporary views based on a query but I don't know how to read from them? Just using the name of the view returns "Table or view not found".

  • 6227 Views
  • 7 replies
  • 2 kudos
Latest Reply
Marra
New Contributor III
  • 2 kudos

No, I'm actually having issues reading from the view in the same session that created it. Using the same view name I get a table or view not found.

  • 2 kudos
6 More Replies
AndriusVitkausk
by New Contributor III
  • 1625 Views
  • 1 replies
  • 1 kudos

Autoloader event vs directory ingestion

For a production work load containing around 15k gzip compressed json files per hour all in a YYYY/MM/DD/HH/id/timestamp.json.gz directoryWhat would be the better approach on ingesting this into a delta table in terms of not only the incremental load...

  • 1625 Views
  • 1 replies
  • 1 kudos
Latest Reply
AndriusVitkausk
New Contributor III
  • 1 kudos

@Kaniz Fatma​ So i've not found a fix for the small file problem using autoloader, seems to struggle really badly against large directories, had a cluster running for 8h stuck on "listing directory" part with no end, cluster seemed completely idle to...

  • 1 kudos
karthikeyanr
by New Contributor II
  • 5454 Views
  • 4 replies
  • 6 kudos

Unable to import .dbc files in Databricks for "Databricks Developer Foundation Capstone"

Hi,I am not able to import .dbc file into Databricks workspace for "Databricks Developer Foundation Capstone". When I click import the error message is displayed. Secondly when I click the github link in the 'Download the capstone' error 404 is displ...

errorgit import error
  • 5454 Views
  • 4 replies
  • 6 kudos
Latest Reply
Atanu
Databricks Employee
  • 6 kudos

Hello @Karthikeyan.r3@cognizant.com R​ I agreed with Hubert. please write to  https://help.databricks.com

  • 6 kudos
3 More Replies
Constantine
by Contributor III
  • 3080 Views
  • 1 replies
  • 1 kudos

Resolved! Can we reuse checkpoints in Spark Streaming?

I am reading data from a Kafka topic, say topic_a. I have an application, app_one which uses Spark Streaming to read data from topic_a. I have a checkpoint location, loc_a to store the checkpoint. Now, app_one has read data till offset 90.Can I creat...

  • 3080 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @John Constantine​,Is not recommended to share the checkpoint with your queries. Every streaming query should have their own checkpoint. If you can to start at the offset 90 in another query, then you can define it when starting your job. You can ...

  • 1 kudos
bhargavi1
by New Contributor II
  • 1560 Views
  • 1 replies
  • 1 kudos
  • 1560 Views
  • 1 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi,Could you share more details on what you have tried? please provide more details.

  • 1 kudos
pbezz
by New Contributor III
  • 7231 Views
  • 5 replies
  • 9 kudos

Visualisation libraries does not work on Databricks

Why is it that certain Python visualisation libraries do not work on Databricks? I am trying to install (via pip) and work with some data visualisation libraries - they work perfectly in a normal Jupyter Notebook but not on a Databricks notebook envi...

  • 7231 Views
  • 5 replies
  • 9 kudos
Latest Reply
pbezz
New Contributor III
  • 9 kudos

No switched to using html widgets.

  • 9 kudos
4 More Replies
Manoj
by Contributor II
  • 15874 Views
  • 13 replies
  • 6 kudos

Resolved! Did any one in the community create Permanent Functions using using python script ? I tried but i am getting the below error, Please advise

Hi Team, When i am trying to register a permanant function i am getting the below error.%sqlCREATE FUNCTION simple_udf AS 'SimpleUdf'  USING JAR '/tmp/SimpleUdf.jar';%sqlselect simple_udf(2)Error Details : com.databricks.backend.common.rpc.Databricks...

  • 15874 Views
  • 13 replies
  • 6 kudos
Latest Reply
Manoj
Contributor II
  • 6 kudos

hi @Werner Stinckens​  @Jose Gonzalez​  @Hubert Dudek​ @Kaniz Fatma​ â€‹Thanks for all the help, Appreciate it. I was able to create permanent functions and use eclipse to create the runnable jar. However, Does anyone have any idea on how to deploy t...

  • 6 kudos
12 More Replies
rbarrero
by New Contributor III
  • 6313 Views
  • 9 replies
  • 7 kudos

Resolved! Error saving changes on Job Cluster

Hello all and thanks.After apply to serving a model, I go to edit corresponding Job Cluster to configure its init_script but when I try to save changes (Confirm and restart) it thrown the following error:Error: Cannot edit cluster 0503-141315-hu3wd4i...

  • 6313 Views
  • 9 replies
  • 7 kudos
Latest Reply
rbarrero
New Contributor III
  • 7 kudos

Sorry for the delay in responding. Finally a partner could fix the problem, he can edit without problems the cluster and add the init_script.Thank you!

  • 7 kudos
8 More Replies
mick042
by New Contributor III
  • 2090 Views
  • 1 replies
  • 3 kudos

Resolved! Spark Executor Rdd.Pipe call not finding file location that exists in Sparkfiles.get()

In a Databricks notebook, I need to run text files (stdin, stdout) through a function from an external library. I have used sparkContext.AddFiles({external_library_name}) to add the external library so that it is available to all executors.when I run...

  • 2090 Views
  • 1 replies
  • 3 kudos
Latest Reply
mick042
New Contributor III
  • 3 kudos

Thanks Kaniz, yes I tried that. Did not work. Falling back on init scripts now and that works.

  • 3 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels