cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Orianh
by Valued Contributor II
  • 3947 Views
  • 4 replies
  • 3 kudos

function does not exist in JVM ERROR

Hello guys, I'm building a python package that return 1 row from DF at a time inside data bricks environment.To improve the performance of this package i used multiprocessing library in python, I have background process that his whole purpose is to p...

function dont exist in JVM error.
  • 3947 Views
  • 4 replies
  • 3 kudos
Latest Reply
dineshreddy
New Contributor II
  • 3 kudos

Using thread instead of processes solved the issue for me

  • 3 kudos
3 More Replies
Paddy_chu
by New Contributor
  • 12596 Views
  • 1 replies
  • 0 kudos

How to restart the kernel on my notebook in databricks?

while installing a python package on my databricks notebook, I kept getting a message saying that: "Note: you may need to restart the kernel using dbutils.library.restartPython() to use updated packages."I've tried restarting my cluster, also detach ...

error message
  • 12596 Views
  • 1 replies
  • 0 kudos
Latest Reply
Evan_MCK
Contributor
  • 0 kudos

dbutils.library.restartPython()Just run this code in the notebook without restarting the cluster or using pip install again. Restarting the cluster erased what you just installed with pip and you are back to square one. Restarting python after the pi...

  • 0 kudos
darthdickhead
by New Contributor III
  • 5277 Views
  • 5 replies
  • 3 kudos

Best way to install and manage a private Python package that has a continuously updating Wheel

I'm trying to setup a Workspace Library that is used internally within our organization. This is a Python package, where the source is available on a private GitHub repository, and not accessible on PyPi or the wider internet / surface web. I managed...

  • 5277 Views
  • 5 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Eshwaran Venkat​ ​​, We haven't heard from you since the last response from @Suteja Kanuri​ ​, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpfu...

  • 3 kudos
4 More Replies
wim_schmitz_per
by New Contributor II
  • 2113 Views
  • 2 replies
  • 2 kudos

Transforming/Saving Python Class Instances to Delta Rows

I'm trying to reuse a Python Package to do a very complex series of parsing binary files into workable data in Delta Format. I have made the first part (binary file parsing) work with a UDF:asffileparser = F.udf(File()._parseBytes,AsfFileDelta.getSch...

  • 2113 Views
  • 2 replies
  • 2 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 2 kudos

Hi, did you try to follow, "Fix it by registering a custom IObjectConstructor for this class."?Also, could you please provide us the full error?

  • 2 kudos
1 More Replies
thushar
by Contributor
  • 2579 Views
  • 4 replies
  • 3 kudos

Resolved! Deploy tar.gz package from private git hub

We created Python package (.tar.gz) and kept it under private git.We can able to connect to that git (using PAT) from the Azure databricks notebook.Our requirement is to install that package from .tar.gz file for that notebook"pip install https://USE...

  • 2579 Views
  • 4 replies
  • 3 kudos
Latest Reply
Rahul_Samant
Contributor
  • 3 kudos

For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/alternatively you can pass the tar.gz using --py-files while submi...

  • 3 kudos
3 More Replies
Labels