cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

wojciech_jakubo
by New Contributor III
  • 5242 Views
  • 7 replies
  • 2 kudos

Question about monitoring driver memory utilization

Hi databricks/spark experts!I have a piece on pandas-based 3rd party code that I need to execute as a part of a bigger spark pipeline. By nature, pandas-based code is executed on driver node. I ran into out of memory problems and started exploring th...

Driver memory cycles_ Busy cluster
  • 5242 Views
  • 7 replies
  • 2 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 2 kudos

Hi @wojciech_jakubo 1. JVM memory will not be utilized for python related activities. 2. In the image we could only see the storage memory. We also have execution memory which would also be the same. Hence I came up with the executor memory to be of ...

  • 2 kudos
6 More Replies
Orianh
by Valued Contributor II
  • 3850 Views
  • 4 replies
  • 3 kudos

function does not exist in JVM ERROR

Hello guys, I'm building a python package that return 1 row from DF at a time inside data bricks environment.To improve the performance of this package i used multiprocessing library in python, I have background process that his whole purpose is to p...

function dont exist in JVM error.
  • 3850 Views
  • 4 replies
  • 3 kudos
Latest Reply
dineshreddy
New Contributor II
  • 3 kudos

Using thread instead of processes solved the issue for me

  • 3 kudos
3 More Replies
blackcoffeeAR
by Contributor
  • 7312 Views
  • 17 replies
  • 5 kudos

How to use/access in a python notebook a scala library installed from JAR file?

I'm using Azure Event Hubs Connector https://github.com/Azure/azure-event-hubs-spark to connect an Even Hub.When I install this library from Maven , then everything works, I can access lib classes using JVM:connection_string = "<connection_string>" s...

2023-02-02 09_30_01-Window
  • 7312 Views
  • 17 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @blackcoffee AR​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 5 kudos
16 More Replies
Joao_DE
by New Contributor III
  • 1390 Views
  • 4 replies
  • 0 kudos

JDBC connection

Hi everyone! I have a question. For a project I need to establish a jdbc connection using spark.read. My question is when does the connection is deleted. That is because I will read multiple tables from that database, so if I could just create a conn...

  • 1390 Views
  • 4 replies
  • 0 kudos
Latest Reply
Joao_DE
New Contributor III
  • 0 kudos

Hi Vidula!I haven´t figure out a solution yet, so any help would be appreciatedThank you!

  • 0 kudos
3 More Replies
James_209101
by New Contributor II
  • 3548 Views
  • 2 replies
  • 4 kudos

Using large dataframe in-memory (data not allowed to be "at rest") results in driver crash and/or out of memory

I'm having trouble working on Databricks with data that we are not allowed to save off or persist in any way. The data comes from an API (which returns a JSON response). We have a scala package on our cluster that makes the queries (almost 6k queries...

  • 3548 Views
  • 2 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @James Held​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 4 kudos
1 More Replies
ivanychev
by Contributor
  • 753 Views
  • 0 replies
  • 1 kudos

How to enable remote JMX monitoring in Databricks?

Adding these optionsEXTRA_JAVA_OPTIONS = ( '-Dcom.sun.management.jmxremote.port=9999', '-Dcom.sun.management.jmxremote.authenticate=false', '-Dcom.sun.management.jmxremote.ssl=false', )is enough in vanilla Apache Spark, but apparently it ...

  • 753 Views
  • 0 replies
  • 1 kudos
dimoobraznii
by New Contributor III
  • 4243 Views
  • 6 replies
  • 6 kudos

Resolved! Autoloader failed

I used autoloader with TriggerOnce = true and ran it for weeks with schedule. Today it broke:The metadata file in the streaming source checkpoint directory is missing. This metadatafile contains important default options for the stream, so the stream...

  • 4243 Views
  • 6 replies
  • 6 kudos
Latest Reply
Deepak_Bhutada
Contributor III
  • 6 kudos

Hi dimoobraznii (Customer)This error comes in streaming when someone makes changes to the streaming checkpoint directory manually or points some streaming type to the checkpoint of some other streaming type. Please check if any changes were made to t...

  • 6 kudos
5 More Replies
Labels