Dear @Louis_Frolio, thankyou for the detailed answer. It certainly cleared things out for me. Totally agree with the suggestion you provided for the long term stability of the production job and I will surely be implementing that. 

About the solutions you provided for the quick test in an interactive environment, I guess testing on a job cluster is the best one to try first. I even tried doing it myself earlier too, but, I could not figure out the way to run a notebook interactively with a job cluster. Job cluster does not appear in the cluster's selection dropdown in the notebook. Would be really great if you could guide me to any resource that explain how can we run notebooks with the job cluster? 
Thankyou!