Multiple concurrent jobs using interactive cluster

[jigar191089]

Hi All,

I have notebook in Databricks. This notebook is executed from azure datafactory pipeline having a databricks notebook activity with linkedservice connected to an interactive cluster.

When multiple concurrent runs of this pipeline are created, I am observing that the notebook job is going in some endless loop.

The command where this happens is as below

import inspect
import json

import sys

sys.path.insert(0, '/dbfs/DataEnabling/Pyspark')

# import user defined libraries
from utils import *
from trigger_processing_framework.insert_trigger import InsertTrigger
from trigger_processing_framework.update_trigger import UpdateTrigger

Later upon restarting the cluster and trying again for the same scenario, the jobs are completing as per expectation.

Any idea what may have went wrong in the first attempt.

Also when multiple runs are created for same notebook, will each run have it own environment state while running or concurrent jobs will interfere with each other since I am using interactive cluster

[nikhilj0421]

Hi @jigar191089, are all the jobs writing to the same location? What is the DBR version you're using? Do you notice any load on the cluster?

Each run will use its own environment, it won't interfere with each other. 

[jigar191089]

@nikhilj0421 , the jobs are writing to different location. DBR is 14.3 ML LTS. Not sure, how to check the load. But if you see the above code it is just import statements.

[nikhilj0421]

@jigar191089, you can monitor the metrics section of the cluster to check the load on the cluster. 
Also, you will see "driver is up but not responsive due to GC" messages in the cluster's event log. 

Can you share the stdout.txt and stderr.txt file when the job gets stuck?

[jigar191089]

Hi @nikhilj0421 I am not able to attach txt file here. This is the screenshot

[nikhilj0421]

Event logs confirm that it isn't because of the driver is under memory pressure. 

Checking the stdout will be very helpful here. Could you please share a screenshot of what you see after Ctrl + C does not work?

Also, are you seeing any library issue in your stderr or stdout? 
 

[jigar191089]

no observation in stderr and stdout. How can I share the log as only attachment of : jpg, gif, png, pdf. are allowed

[nikhilj0421]

Do you see any attachment option?

[nikhilj0421]

What libraries are you installing in your cluster?

[jigar191089]

There are few custom build and then few libraries available on PyPi

[nikhilj0421]

Can you share the screenshots of the pypi library installed on your cluster?

[jigar191089]

Do note. This libraries are installed using the cluster init script