Hi - just in case anyone else is still experiencing this issue - please see below how I fixed this...Go to your Azure Portal 'Activity log' and then look for any errors whilst running the Databricks Pipeline...I received this error in my Activity Log...
