Hi - just in case anyone else is still experiencing this issue - please see below how I fixed this...

Go to your Azure Portal 'Activity log' and then look for any errors whilst running the Databricks Pipeline...

I received this error in my Activity Log: "Create or Update Virtual Machine - Failed" Operation could not be completed as it results in exceeding the approved standardFSFamily Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 10, Current Usage: 8, Additional Required: 8, (Minimum) New Limit Required: 16. Submit a request for Quota increase at [***URL***] by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://disq.us/url?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fazure-supportability%2Fper-vm-quota-requests%3AvJyPQQt21s9--6dqZy4z2vOT8JA&cuid=5011031

I then requested an increase for the following quotas: -

Standard FS Family vCPUs increased from 10 to 50

Standard DSv2 Family vCPUs increased from 10 to 50

And now re-run the pipeline - hopefully, this fixes the issue for you too 🙂