- 16596 Views
- 2 replies
- 4 kudos
Resolved! Failure starting repl. Try detaching and re-attaching the notebook
I just started my manual cluster this morning in the production environment to run some code and it isn't executing and giving me the error "Failure starting repl. Try detaching and re-attaching the notebook.".What can I do to solve this?I have tried...
- 16596 Views
- 2 replies
- 4 kudos
- 4 kudos
Just in case anyone needs to know how to solve this in the future.Apparently one of my clusters was suddenly having library compatibility issues. Mainly between pandas,numpy and pyarrow.So I fixed this by forcing specific versions in my global init s...
- 4 kudos
- 2476 Views
- 2 replies
- 0 kudos
Resolved! Runtime 11.3 LTS not working in my production
Hello,I have a cluster with Runtime 11.3 LTS in my production. Whenever I start this up and try to run my notebooks it's giving me error: Failure starting repl. Try detaching and re-attaching the notebook. I have a cluster with the same Runtime in my...
- 2476 Views
- 2 replies
- 0 kudos
- 0 kudos
Just in case anyone needs to know how to solve this in the future.Apparently one of my clusters was suddenly having library compatibility issues. Mainly between pandas,numpy and pyarrow.So I fixed this by forcing specific versions in my global init s...
- 0 kudos
- 1444 Views
- 2 replies
- 0 kudos
drop duplicate in 500B records
I’m trying to drop duplicate in a DF where I have 500B records I’m trying to delete based on multiple columns but this process it’s takes 5h, I try lot of things that available on internet but nothing is works for me.my code is like this.df_1=spark....
- 1444 Views
- 2 replies
- 0 kudos
- 0 kudos
Drop the duplicates from the df_1 and df_2 first and then do the join.If the join is just a city code, then most likely you know which rows in df_2 and in df_1 will give you the duplicates in df_join. So drop in df_1 and drop in df_2 instead of df_jo...
- 0 kudos
- 6441 Views
- 1 replies
- 0 kudos
Resolved! Escaping $ (dollar sign) in a regex backreference in notebook (so not seen as a parameter)
I am trying to do a regular expression replace in a Databricks notebook.The following query works fine in a regular query (i.e. not running it in a cell in a notebook): select regexp_replace('abcd', '^(.+)c(.+)$', '$1_$2') --normally outputs ab_d H...
- 6441 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, just put a backslash before $ as an escape character:
- 0 kudos
- 1037 Views
- 1 replies
- 0 kudos
ISSUE: PySpark task exception handling on "Shared Compute" cluster
I am experiencing an issue with a PySpark job that behaves differently depending on the compute environment in Databricks. And this is blocking us from deploying the job into the PROD environment for our planned release.Specifically:- When running th...
- 1037 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @geronimo_signol ,Recently, other user has reported similar behavior on shared clusters, and both issues seem to be related to Spark Connect.To verify whether your cluster is using Spark Connect, please run the following code in your notebook: pri...
- 0 kudos
- 2129 Views
- 3 replies
- 0 kudos
Explaining the explain plan
Hi All,I am new to Databricks and have recently started exploring databricks' explain plans to try and understand how the queries are executed (and eventually tune them as needed).There are some things that I can somehow "guess" based on what I know ...
- 2129 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @annetemplon ,There are plenty of resources about this topic but they are scattered all over internet I like below videos, pretty informative:https://m.youtube.com/watch?v=99fYi2mopbshttps://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&u...
- 0 kudos
- 17709 Views
- 4 replies
- 8 kudos
Resolved! notebook for SFTP server connectivity without password.
I am trying to develop some script using python to access an sftp server without password and all valid public/private keys in a notebook. However I am not getting any such example. All examples has a password in it. Can I get some help?
- 17709 Views
- 4 replies
- 8 kudos
- 8 kudos
https://stackoverflow.com/questions/58562744/how-to-upload-text-file-to-ftp-from-databricks-notebook this example looks good to me. or may be try using data libs. https://www.cdata.com/kb/tech/sftp-jdbc-azure-databricks.rst
- 8 kudos
- 1808 Views
- 1 replies
- 2 kudos
Vulnerability in the Guava dependency of the Databricks jdbc driver
Good afternoon I want to report that the JDBC driver incorporates a version of com.google.guava:guava that has two vulnerabilities. Image attached. Could the dependency be updated?
- 1808 Views
- 1 replies
- 2 kudos
- 2 kudos
- 2 kudos
- 3241 Views
- 2 replies
- 1 kudos
Resolved! Dynamic Python UDF in unity catalog
Hi Team I am trying to create a python UDF which I want to use for column masking. This function will take 2 input parameters(column name and groups name) and return the column value if user is part of group otherwise return masked value. I wrote fol...
- 3241 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Dnirmania, You could achieve something similar using this UDF:%sql CREATE OR REPLACE FUNCTION ryanlakehouse.default.column_masking(column_value STRING, groups_str String) RETURNS STRING LANGUAGE SQL COMMENT 'Return the column value if use...
- 1 kudos
- 4366 Views
- 2 replies
- 1 kudos
Workflow file arrival trigger - does it apply to overwritten files?
I am exploring the use of the "file arrival" trigger for a workflow for a use case I am working on. I understand from the documentation that it checks every minute for new files in an external location, then initiates the workflow when it detects a ...
- 4366 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @mvmiller The "file arrival" trigger for a workflow considers the name of the file,when the same name file was overwritten the workflow didnt triggerred.hope I answered your question!
- 1 kudos
- 2902 Views
- 3 replies
- 0 kudos
One worker is one executor and one physical node
Hi team,Seems in Databricks, instead of like running Spark jobs on a k8s cluster, when a workflow running on a Job Compute/Cluster or instance pool, one physical node can only have one executor. Is this understanding right? If that is true, that mean...
- 2902 Views
- 3 replies
- 0 kudos
- 0 kudos
It can change the number of executors dynamically if I am not mistaken.But maybe Databricks has hammered the 1-1 ratio in stone, something to test I'd say.
- 0 kudos
- 7324 Views
- 5 replies
- 1 kudos
DLT - runtime parameterisation of execution
I have started to use DLT in a prototype framework and I now face the below challenge for which any help would be appreciated.First let me give a brief context:I have metadata sitting in a .json file that I read as the first task and put it into a lo...
- 7324 Views
- 5 replies
- 1 kudos
- 1 kudos
@Retired_mod Can you please provide some reference to REST API approach? I do not see that available on the docs. TIA
- 1 kudos
- 1073 Views
- 1 replies
- 0 kudos
how to write data frame API
how to write data frame API
- 1073 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @shivkumar , I hope this message finds you well. I didn't fully understand your question, but to write a DataFrame you can follow the steps in the link below:https://docs.databricks.com/en/getting-started/dataframes.html#tutorial-load-and-trans...
- 0 kudos
- 5797 Views
- 8 replies
- 1 kudos
Resolved! FileNotFoundError while reading PDF file in Databricks from DBFS location
I am trying to read a PDF file from DBFS location in Databricks using PyPDF2.PdfFileReader but it's throwing error that file doesn't existBut the file exists in the path, refer below screenshotCan anyone please suggest what is wrong in this?
- 5797 Views
- 8 replies
- 1 kudos
- 1 kudos
@sahil07, It seems that with your current setup, you can't read from DBFS using vanilla Python. I've ran some tests and managed to reproduce the error and solve it by copying the file in DBFS to the local file system of the driver node using dbutils....
- 1 kudos
- 3383 Views
- 1 replies
- 0 kudos
Error deploying DLT DAB: Validation failed for cluster_type, the value must be dlt (is "job")
Hello.I'm getting a cluster validation error while trying to deploy a DLT pipeline via DAB.See attached screenshots for config and error.Hoping someone has run into this before and can guide me.Thanks.
- 3383 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @ahab,Were you able to solve this issue, if yes would you mind sharing your finds If not, looking at your cluster policy, I'm seeing that you are using dlt as the cluster type which is the correct needed type for DLT pipelines, but in the resourc...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
