- 6165 Views
- 4 replies
- 2 kudos
Resolved! Unable to delete a DBFS folder
Hello everyone,I've created by error a DBFS folder named : ${env]But when I run this command :dbutils.fs.rm("/mnt/${env]")It returns me this error : java.net.URISyntaxException: Illegal character in path at index 12: /mnt/$%7Benv]How can I do please ...
- 6165 Views
- 4 replies
- 2 kudos
- 5388 Views
- 2 replies
- 3 kudos
Why does my Notebook fails if I try to load a function from another Notebook in Repos in Databricks
My function in func.pydef lower_events(df): return df.withColumn("event",f.lower(f.col("event")))My main notebook import pyspark.sql.functions as f from pyspark.sql.functions import udf, col, lower import sys sys.path.append("..") from folder.func...
- 5388 Views
- 2 replies
- 3 kudos
- 3 kudos
@Kaniz Fatma​ https://community.databricks.com/s/question/0D58Y00008ouo6xSAA/how-to-fetch-environmental-variables-saved-in-one-notebook-into-another-notebook-in-databricks-repos-and-notebooksCan you please look into this
- 3 kudos
- 4867 Views
- 2 replies
- 2 kudos
Resolved! Importing python function with spark.read.jdbc in to Repos
Hi all!Before we used Databricks Repos we used the run magic to run various utility python functions from one notebook inside other notebooks, fex like reading from a jdbc connections. We now plan to switch to repos to utilize the fantastic CI/CD pos...
- 4867 Views
- 2 replies
- 2 kudos
- 2 kudos
Thats...odd. I was sure I had tried that, but now it works somehow. I guess it has to be that now I did it with double quotation marks. Thanks anyway! Works like a charm.
- 2 kudos
- 11457 Views
- 7 replies
- 2 kudos
Read temporary views in SQL Analytics
I'm having issues trying to read temporary views in SQL Analytics module. Ive managed to create temporary views based on a query but I don't know how to read from them? Just using the name of the view returns "Table or view not found".
- 11457 Views
- 7 replies
- 2 kudos
- 2 kudos
No, I'm actually having issues reading from the view in the same session that created it. Using the same view name I get a table or view not found.
- 2 kudos
- 2744 Views
- 1 replies
- 1 kudos
Autoloader event vs directory ingestion
For a production work load containing around 15k gzip compressed json files per hour all in a YYYY/MM/DD/HH/id/timestamp.json.gz directoryWhat would be the better approach on ingesting this into a delta table in terms of not only the incremental load...
- 2744 Views
- 1 replies
- 1 kudos
- 1 kudos
@Kaniz Fatma​ So i've not found a fix for the small file problem using autoloader, seems to struggle really badly against large directories, had a cluster running for 8h stuck on "listing directory" part with no end, cluster seemed completely idle to...
- 1 kudos
- 8435 Views
- 4 replies
- 6 kudos
Unable to import .dbc files in Databricks for "Databricks Developer Foundation Capstone"
Hi,I am not able to import .dbc file into Databricks workspace for "Databricks Developer Foundation Capstone". When I click import the error message is displayed. Secondly when I click the github link in the 'Download the capstone' error 404 is displ...
- 8435 Views
- 4 replies
- 6 kudos
- 6 kudos
Hello @Karthikeyan.r3@cognizant.com R​ I agreed with Hubert. please write to https://help.databricks.com
- 6 kudos
- 6098 Views
- 1 replies
- 1 kudos
Resolved! Can we reuse checkpoints in Spark Streaming?
I am reading data from a Kafka topic, say topic_a. I have an application, app_one which uses Spark Streaming to read data from topic_a. I have a checkpoint location, loc_a to store the checkpoint. Now, app_one has read data till offset 90.Can I creat...
- 6098 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @John Constantine​,Is not recommended to share the checkpoint with your queries. Every streaming query should have their own checkpoint. If you can to start at the offset 90 in another query, then you can define it when starting your job. You can ...
- 1 kudos
- 2541 Views
- 1 replies
- 1 kudos
- 2541 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi,Could you share more details on what you have tried? please provide more details.
- 1 kudos
- 11303 Views
- 5 replies
- 9 kudos
Visualisation libraries does not work on Databricks
Why is it that certain Python visualisation libraries do not work on Databricks? I am trying to install (via pip) and work with some data visualisation libraries - they work perfectly in a normal Jupyter Notebook but not on a Databricks notebook envi...
- 11303 Views
- 5 replies
- 9 kudos
- 28427 Views
- 13 replies
- 6 kudos
Resolved! Did any one in the community create Permanent Functions using using python script ? I tried but i am getting the below error, Please advise
Hi Team, When i am trying to register a permanant function i am getting the below error.%sqlCREATE FUNCTION simple_udf AS 'SimpleUdf' USING JAR '/tmp/SimpleUdf.jar';%sqlselect simple_udf(2)Error Details : com.databricks.backend.common.rpc.Databricks...
- 28427 Views
- 13 replies
- 6 kudos
- 6 kudos
hi @Werner Stinckens​ @Jose Gonzalez​ @Hubert Dudek​ @Kaniz Fatma​ ​Thanks for all the help, Appreciate it. I was able to create permanent functions and use eclipse to create the runnable jar. However, Does anyone have any idea on how to deploy t...
- 6 kudos
- 9061 Views
- 9 replies
- 7 kudos
Resolved! Error saving changes on Job Cluster
Hello all and thanks.After apply to serving a model, I go to edit corresponding Job Cluster to configure its init_script but when I try to save changes (Confirm and restart) it thrown the following error:Error: Cannot edit cluster 0503-141315-hu3wd4i...
- 9061 Views
- 9 replies
- 7 kudos
- 7 kudos
Sorry for the delay in responding. Finally a partner could fix the problem, he can edit without problems the cluster and add the init_script.Thank you!
- 7 kudos
- 3402 Views
- 1 replies
- 3 kudos
Resolved! Spark Executor Rdd.Pipe call not finding file location that exists in Sparkfiles.get()
In a Databricks notebook, I need to run text files (stdin, stdout) through a function from an external library. I have used sparkContext.AddFiles({external_library_name}) to add the external library so that it is available to all executors.when I run...
- 3402 Views
- 1 replies
- 3 kudos
- 3 kudos
Thanks Kaniz, yes I tried that. Did not work. Falling back on init scripts now and that works.
- 3 kudos
- 6659 Views
- 3 replies
- 2 kudos
Resolved! Automation creation of Databricks Top Level Repositories
I'm looking to automate the creation of Top Level repositories in Databricks however isn't possible using cli or API if this repo is private repository(Azure DevOps Repository) because require setup the token in user setting.databricks repos create \...
- 6659 Views
- 3 replies
- 2 kudos
- 2 kudos
We have API availabe for repos https://docs.databricks.com/dev-tools/api/latest/repos.html#operation/get-repos also we are currently supporting SP. Step 1: As an admin Create a Service PrincipalUse this API SCIM API 2.0 (ServicePrincipals) | Databric...
- 2 kudos
- 7128 Views
- 7 replies
- 6 kudos
Unicode field separator to create unamanged table in databricks for csv file
We are getting \u318a (ㆊ) separated csv file. We want to create unmanaged table in databricks, Here is the table creation script.create table IF NOT EXISTS db_test_raw.t_data_otc_poc (`caseidt` String,`worktype` String,`doctyp` String,`brand` String,...
- 7128 Views
- 7 replies
- 6 kudos
- 6 kudos
sep "\u318a"delimeter " \x318a"sep " \x318a"Try to use sep instead or/and x instead.
- 6 kudos
- 5386 Views
- 2 replies
- 4 kudos
Resolved! Why can't I create new jobs? ("You are not entitled to run this type of task...")
This morning I encountered an issue when trying to create a new job using the Workflows UI (in browser). Never had this issue before.The error message that appears is:"You are not entitled to run this type of task, please contact your Databricks admi...
- 5386 Views
- 2 replies
- 4 kudos
- 4 kudos
@Kaniz Fatma​ @Philip Nord​, thanks!I was able to do what I needed by cloning an existing job & modifying. It's fine as a temporary fix for now.Thanks again for the response-- good to know you're aware of it & this isn't anything on my end.
- 4 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
