Hi All,

I have notebook in Databricks. This notebook is executed from azure datafactory pipeline having a databricks notebook activity with linkedservice connected to an interactive cluster.

When multiple concurrent runs of this pipeline are created, I am observing that the notebook job is going in some endless loop.

The command where this happens is as below

import inspect
import json

import sys

sys.path.insert(0, '/dbfs/DataEnabling/Pyspark')

# import user defined libraries
from utils import *
from trigger_processing_framework.insert_trigger import InsertTrigger
from trigger_processing_framework.update_trigger import UpdateTrigger

Later upon restarting the cluster and trying again for the same scenario, the jobs are completing as per expectation.

Any idea what may have went wrong in the first attempt.

Also when multiple runs are created for same notebook, will each run have it own environment state while running or concurrent jobs will interfere with each other since I am using interactive cluster