cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Paddy_chu
by • New Contributor
  • 22330 Views
  • 3 replies
  • 3 kudos

How to restart the kernel on my notebook in databricks?

while installing a python package on my databricks notebook, I kept getting a message saying that: "Note: you may need to restart the kernel using dbutils.library.restartPython() to use updated packages."I've tried restarting my cluster, also detach ...

error message
Data Engineering
Reply
  • 22330 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
johnb1
Contributor
  • 3 kudos

@Evan_MCK Follow-up question:When other notebooks run Python code on the same cluster, will those runs be aborted when dbutils.library.restartPython() is called?

  • 3 kudos
2 More Replies
Orianh
by • Valued Contributor II
  • 5567 Views
  • 4 replies
  • 3 kudos

function does not exist in JVM ERROR

Hello guys, I'm building a python package that return 1 row from DF at a time inside data bricks environment.To improve the performance of this package i used multiprocessing library in python, I have background process that his whole purpose is to p...

function dont exist in JVM error.
Data Engineering
Reply
  • 5567 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
dineshreddy
New Contributor III
  • 3 kudos

Using thread instead of processes solved the issue for me

  • 3 kudos
3 More Replies
darthdickhead
by • New Contributor III
  • 8680 Views
  • 4 replies
  • 3 kudos

Best way to install and manage a private Python package that has a continuously updating Wheel

I'm trying to setup a Workspace Library that is used internally within our organization. This is a Python package, where the source is available on a private GitHub repository, and not accessible on PyPi or the wider internet / surface web. I managed...

Data Engineering
Reply
  • 8680 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Eshwaran Venkat​ You can use the Databricks CLI to automate the process of cloning the private GitHub repository and building/uploading the Python package to DBFS as a wheel file. You can schedule this process to run periodically, such as once a day...

  • 3 kudos
3 More Replies
wim_schmitz_per
by • New Contributor II
  • 3608 Views
  • 2 replies
  • 2 kudos

Transforming/Saving Python Class Instances to Delta Rows

I'm trying to reuse a Python Package to do a very complex series of parsing binary files into workable data in Delta Format. I have made the first part (binary file parsing) work with a UDF:asffileparser = F.udf(File()._parseBytes,AsfFileDelta.getSch...

Data Engineering
Reply
  • 3608 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 2 kudos

Hi, did you try to follow, "Fix it by registering a custom IObjectConstructor for this class."?Also, could you please provide us the full error?

  • 2 kudos
1 More Replies
thushar
by • Contributor
  • 3782 Views
  • 4 replies
  • 3 kudos

Resolved! Deploy tar.gz package from private git hub

We created Python package (.tar.gz) and kept it under private git.We can able to connect to that git (using PAT) from the Azure databricks notebook.Our requirement is to install that package from .tar.gz file for that notebook"pip install https://USE...

Data Engineering
Reply
  • 3782 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Rahul_Samant
Contributor
  • 3 kudos

For installing the package using pip you need to package the repo using setup.py. check this link for more details https://packaging.python.org/en/latest/tutorials/packaging-projects/alternatively you can pass the tar.gz using --py-files while submi...

  • 3 kudos
3 More Replies
Labels