When using databricks-connect how can you monitor the progress? Ideally, we want something similar to what you get in the Databricks notebook, i.e., information about the jobs/stages. We are using Python.
Yes, exactly that: the progress information about the job(s) and their stages that the query triggered. We really need this information to be displayed in the application.
How are you supposed to do this when using the setup described here: https://www.databricks.com/blog/data-exfiltration-protection-with-azure-databricks Would you need to deploy a NAT gateway together with the Azure Firewall then?