My PoV please, 

We use Databricks REST API; and we have faced 401 or Azure Front door related auth issues. Like you said, we use retry mechanism. Runtime errors are recorded and retry attempts are made if the tasks are idempotent; otherwise user intervention is required. 

We plan not to remove this retry process irrespective of Azure/Databricks sporadic issues are fixed at their side.