Thanks for the suggestion, it turns out the problem is with the setup of the compute cluster i.e. the cluster I have been using is set up correctly and can access the metastore but isnt ML runtime and when I set up my own ML runtime cluster I cannot configure it to see the metastore.  I think this means the metastore is external to databricks  and I have asked the admin to either set up an ML cluster with the right authentication or allow me to create one.  Long story short I cant do as you suggest because attaching the notebook to an ML cluster also breaks the connection to the SQL query.