cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SS2
by • Valued Contributor
  • 1930 Views
  • 2 replies
  • 1 kudos

Spark out of memory error.

Sometimes in Databricks you can see the out of memory error then in that case you can change the cluster size. As per requirement to resolve the issue.

Data Engineering
Reply
  • 1930 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @S S​,Could you provide more details on your issue? for example, error stack traces, code snippet, etc. We will be able to help you if you share more details

  • 1 kudos
1 More Replies
SS2
by • Valued Contributor
  • 5824 Views
  • 4 replies
  • 3 kudos

Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

Spark out of memory error.You can resolve this error by increasing the size of cluster in Databricks.

Data Engineering
Reply
  • 5824 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
DK03
Contributor
  • 3 kudos

Adding some more points to @karthik p​ 's answer.Use kryo serializer instead of java serializer.Use an optimised garbage collector such as G1GC.Use partitioning wisely on a field.

  • 3 kudos
3 More Replies
Bujji
by • New Contributor II
  • 5125 Views
  • 1 replies
  • 3 kudos

How to resolve our of memory error?

Hi, I am working as azure support engineerI found this error while I am checking the pipeline failure, and showing below error"org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 72403.0 failed 4 times, most recent fail...

Data Engineering
Reply
  • 5125 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Pat
Honored Contributor III
  • 3 kudos

Hi @mahesh bmk​ ,It would be nice to see the sql_query.is there some window function used? You might try to run this on bigger cluster.

  • 3 kudos
potluri
by • New Contributor II
  • 3196 Views
  • 1 replies
  • 1 kudos

Resolved! Cluster frequently crashing

Cluster crashing, prompting me to use a different cluster or restart the cluster. Previously worked fine for the same code

Data Engineering
Reply
  • 3196 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @potluri​ ,What kind of cluster care you using? Is it an interactive cluster or a job cluster? what is the error message you are getting? The following KB article could help you to find the cause and the solution to your problem. Please check the ...

  • 1 kudos
Labels