Hi @manivannan p , Please refer to this S.O thread.
Also,
Problem
A Databricks job fails because the job requires a library that is not yet installed, causing Import errors.
Cause
The error occurs because the job starts running before the required libraries are installed.
If you run a job on a cluster in either of the following situations, the cluster can experience a delay in installing libraries:
- When you start an existing cluster with libraries in the terminated state.
- When you start a new cluster that uses a shared library (installed on all clusters).
Solution