Job running time too long

galzamo — Wed, 05 Jun 2024 08:09:14 GMT

Hi all,

I'm doing my first data jobs.

I create one job that consists of 4 other jobs.

Yesterday I ran the 4 jobs separately and it worked fine (about half hour)-

today I ran the big job, and the 4 jobs is running for 2 hours (and still running),

Why is that happening? I'm using the same compute

Thanks!

Re: Job running time too long

anardinelli — Wed, 05 Jun 2024 14:49:47 GMT

Hello @galzamo how are you?

You can check on the SparkUI for long running stages that might give you a clue where it's spending the most time on each task. Somethings can be the reason:

1. Increase of data and partitions on your source data

2. Cluster concurrency (if you're using a shared cluster with other users)

3. Network and connection issues when connecting to external data sources

If you can share more of your job and the spark logs, we can help you to check.

Best,

Alessandro

topic Re: Job running time too long in Data Engineering

Job running time too long

Re: Job running time too long