cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
nadia
by • New Contributor II
  • 18619 Views
  • 3 replies
  • 2 kudos

Resolved! Executor heartbeat timed out

Hello, I'm trying to read a table that is located on Postgreqsl and contains 28 million rows. I have the following result:"SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in sta...

Data Engineering
Reply
  • 18619 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
JunYang
New Contributor III
  • 2 kudos

Please also review the Spark UI to see the failed Spark job and Spark stage. Please check on the GC time and data spill to memory and disk. See if there is any error in the failed task in the Spark stage view. This will confirm data skew or GC/memory...

  • 2 kudos
2 More Replies
Data_Analytics1
by • Contributor III
  • 15660 Views
  • 8 replies
  • 2 kudos

TimeoutException: Futures timed out after [5 seconds]. I am getting this error while running few parallel jobs at an interval of 5 minutes.

java.util.concurrent.TimeoutException: Futures timed out after [5 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263) at scala.concurrent.Await$.$...

Data Engineering
Reply
  • 15660 Views
  • 8 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Mahesh Chahare​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us s...

  • 2 kudos
7 More Replies
arul_parthiban
by • New Contributor
  • 660 Views
  • 0 replies
  • 0 kudos

How to customize the result of COPY INTO command?

COPY INTO statement produces results in following format, which is more similar to INSERT INTO statement results; also it's a summary of all the files loaded. Is there a way to customize in way that it produces the detailed results at file level?

image
Data Engineering
Reply
  • 660 Views
  • 0 replies
  • 0 kudos
data_boy_2022
by • New Contributor III
  • 2285 Views
  • 2 replies
  • 0 kudos

Resolved! Writing transformed DataFrame to a persistent table is unbearable slow

I want to transform a DF with a simple UDF. Afterwards I want to store the resulting DF in a new table (see code below)key = "test_key"   schema = StructType([ StructField("***", StringType(), True), StructField("yyy", StringType(), True), StructF...

Data Engineering
Reply
  • 2285 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Vidula
Honored Contributor
  • 0 kudos

Hello @Jan R​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
Tahseen0354
by • Valued Contributor
  • 19986 Views
  • 8 replies
  • 5 kudos

Resolved! Getting "Job aborted due to stage failure" SparkException when trying to download full result

I have generated a result using SQL. But whenever I try to download the full result (1 million rows), it is throwing SparkException. I can download the preview result but not the full result. Why ? What happens under the hood when I try to download ...

Data Engineering
Reply
  • 19986 Views
  • 8 replies
  • 5 kudos
View Replies
Latest Reply
rpshgupta
New Contributor III
  • 5 kudos

I am also having this issue again and again. I really want to understand what can we do to avoid this?

  • 5 kudos
7 More Replies
Labels