Thanks for your responses.

 @Retired_mod it is not code or configuration issue.  The job runs successfully for other runs, that run with issue was an exceptional case.

 @ajay yes it sure looks like databricks internal issue. Was wondering if anyone has faced similiar issue