cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sandeep91
by New Contributor III
  • 4902 Views
  • 5 replies
  • 2 kudos

Resolved! Databricks Job: Package Name and EntryPoint parameters for the Python Wheel file

I have created Python wheel file with simple file structure and uploaded into cluster library and was able to run the packages in Notebook but, when I am trying to create a Job using python wheel and provide the package name and run the task it fails...

image
  • 4902 Views
  • 5 replies
  • 2 kudos
Latest Reply
AndréSalvati
New Contributor III
  • 2 kudos

There you can see a complete template project with (the new!!!) Databricks Asset Bundles tool and a python wheel task. Please, follow the instructions for deployment.https://github.com/andre-salvati/databricks-template

  • 2 kudos
4 More Replies
kaileena
by New Contributor
  • 823 Views
  • 2 replies
  • 0 kudos

cannot install RMySQL "there is no package called ‘RMySQL’

cannot install RMySQL on databricks. i tried:install.packages("RMySQL")i got the error:Installing package into ‘/local_disk0/.ephemeral_nfs/envs/rEnv-c677bc4c-e6a3-40df-a5ab-bfd5d277e0c0’ (as ‘lib’ is unspecified) Warning: unable to access index for ...

  • 823 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @miru miro​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 0 kudos
1 More Replies
tessaickx
by New Contributor III
  • 1632 Views
  • 3 replies
  • 3 kudos

Using ipywidgets latest versions

Hello everyone,I upgraded my cluster to DBR 13.0, which comes with ipywidgets version 7.7.2 installed.However, I want to use the TagsInput widget, which is new since version 8.0.4.If i upgrade the ipywidgets package to version 8.0.4, none of the widg...

  • 1632 Views
  • 3 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Tessa Ickx​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 3 kudos
2 More Replies
-werners-
by Esteemed Contributor III
  • 1210 Views
  • 3 replies
  • 22 kudos

Resolved! Package cells (scala), who uses them?

So I was wondering who uses package cells in scala?We have this library (jar) which has some useful functions we use all over the place. But that's about it. So I think we can do the same thing without a jar but with package cells.But I never hear ...

  • 1210 Views
  • 3 replies
  • 22 kudos
Latest Reply
Anonymous
Not applicable
  • 22 kudos

Hi @Werner Stinckens​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.T...

  • 22 kudos
2 More Replies
Ross
by New Contributor II
  • 1111 Views
  • 1 replies
  • 0 kudos

Failed R install package of survminer in Databricks 10.4 LTS

I am trying to install the survminer package but I get a non-zero exit status. It may be due to the jpeg package which is a pre-requisite but this also fails when installing independently.install.packages("survminer", repos = "https://cran.microsoft....

  • 1111 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@Ross Hamilton​ - Please follow the below steps in the given orderRun the below init script in an isolated notebook and add the init script to the issue cluster > Advanced options > Init Scripts%python dbutils.fs.put("/tmp/test/init_script.sh",""" #...

  • 0 kudos
Sulfikkar
by Contributor
  • 8917 Views
  • 5 replies
  • 3 kudos

Resolved! install a custom python package from azure devops artifact to databricks cluster

I am trying to install a package which was uploaded into the azure devops artifact into the databricks cluster by using pip.conf. Basically below are the steps I followed.(step 1 : install in local IDE)Uploaded the package to azure devops feed using ...

  • 8917 Views
  • 5 replies
  • 3 kudos
Latest Reply
Sulfikkar
Contributor
  • 3 kudos

Thanks for your time @Debayan Mukherjee​  and @Kaniz Fatma​ . We have figured out the issue along with the infra team that we had to do a public ip whitelisting of the databricks clusters in azure.I have checked the ip adress from the Spark cluster U...

  • 3 kudos
4 More Replies
BorislavBlagoev
by Valued Contributor III
  • 2339 Views
  • 3 replies
  • 5 kudos

Resolved! Get package from Nexus repo.

I want to receive a package from Nexus repo both in notebook and job. If anyone has experience with this, please answer me here!

  • 2339 Views
  • 3 replies
  • 5 kudos
Latest Reply
Kaniz
Community Manager
  • 5 kudos

Hi @Borislav Blagoev​ , Just a friendly follow-up. Do you still need help, or does the above response help you to find the solution? Please let us know.

  • 5 kudos
2 More Replies
sarvesh242
by Contributor
  • 6394 Views
  • 7 replies
  • 2 kudos

Resolved! java.lang.NoSuchMethodError in databricks

I have created a package in scala. Now, I am calling a method from that package and using it in my notebook. During run time, it throws me an error java.lang.NoSuchMethodError. The method exists in the package but still, I am getting this error. Plea...

  • 6394 Views
  • 7 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

@sarvesh gakhar​ , Just a friendly follow-up. Do you still need help? Please let us know.

  • 2 kudos
6 More Replies
sarvesh242
by Contributor
  • 1012 Views
  • 2 replies
  • 2 kudos

Resolved! java.lang.NoSuchMethodError in databricks

I have created a package. Now I am calling a method from this package in my notebook but it is throwing me java.lang.NoSuchMethodError in databricks. The method exists in the package. Can you please guide me regarding the same.Thanks!

  • 1012 Views
  • 2 replies
  • 2 kudos
Latest Reply
sarvesh242
Contributor
  • 2 kudos

Hi! I am sharing the error stack with you. I can't share the code with you due to confidentiality of the code. Can you please guide me ?java.lang.NoSuchMethodError: com.iig.utils.common.IIGCommonConstants$.flowProperties()Ljava/lang/String; at com.ii...

  • 2 kudos
1 More Replies
thushar
by Contributor
  • 2481 Views
  • 4 replies
  • 3 kudos

Resolved! Deploy tar.gz package from private git hub

We created Python package (.tar.gz) and kept it under private git.We can able to connect to that git (using PAT) from the Azure databricks notebook.Our requirement is to install that package from .tar.gz file for that notebook"pip install https://USE...

  • 2481 Views
  • 4 replies
  • 3 kudos
Latest Reply
Rahul_Samant
Contributor
  • 3 kudos

For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/alternatively you can pass the tar.gz using --py-files while submi...

  • 3 kudos
3 More Replies
Itachi_Naruto
by New Contributor II
  • 5780 Views
  • 7 replies
  • 2 kudos

Resolved! hdbscan package error

I try to import **hdbscan** but it throws this following error/databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py in import_patch(name, globals, locals, fromlist, level) 156 # Import the desired module. ...

  • 5780 Views
  • 7 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Rajamannar Aanjaram​ , It looks like there's a compatibility issue with the hdbscan library.You may check out the Github issue which addresses a similar issue.In case the above Github issue doesn't solve your issue, I would request to open a new ...

  • 2 kudos
6 More Replies
JonHMDavis
by New Contributor II
  • 2435 Views
  • 5 replies
  • 4 kudos

Resolved! Graphframes not importing on Databricks 9.1 LTS ML

Is Graphframes for python meant to be installed by default on Databricks 9.1 LTS ML? Previously I was running the attached python command on 7.3 LTS ML with no issue, however now I am getting "no module named graphframes" when trying to import the pa...

  • 2435 Views
  • 5 replies
  • 4 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 4 kudos

Hm I thought the same thing as the docs state "The GraphFrames package is included in Databricks Runtime for Machine Learning".But that could have changed...

  • 4 kudos
4 More Replies
raymund
by New Contributor III
  • 2204 Views
  • 7 replies
  • 5 kudos

Resolved! Why adding the package 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1' failed in runtime 9.1.x-scala2.12 but was successful using runtime 8.2.x-scala2.12 ?

Using Databricks spark submit job, setting new cluster1] "spark_version": "8.2.x-scala2.12" => OK, works fine2] "spark_version": "9.1.x-scala2.12" => FAIL, with errorsException in thread "main" java.lang.ExceptionInInitializerError at com.databricks...

  • 2204 Views
  • 7 replies
  • 5 kudos
Latest Reply
raymund
New Contributor III
  • 5 kudos

this has been resolved by adding the following spark_conf (not thru --conf) "spark.hadoop.fs.file.impl": "org.apache.hadoop.fs.LocalFileSystem"example:------"new_cluster": { "spark_version": "9.1.x-scala2.12", ... "spark_conf": { "spar...

  • 5 kudos
6 More Replies
Anonymous
by Not applicable
  • 1862 Views
  • 3 replies
  • 1 kudos

What is the difference between mlflow projects and mlflow model?

 They both seem to package it. When should one use one over the other?

  • 1862 Views
  • 3 replies
  • 1 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 1 kudos

One thing I think it's useful to point out for Databricks users is that you would typically not use MLflow Projects to describe execution of a modeling run. You would just use MLflow directly in Databricks and use Databricks notebooks to manage code ...

  • 1 kudos
2 More Replies
Labels