Monitor progress when using databricks-connect

[kyrrewk]

When using databricks-connect how can you monitor the progress? Ideally, we want something similar to what you get in the Databricks notebook, i.e., information about the jobs/stages. We are using Python.

[Walter_C]

When you refer to progress, you mean that during the notebook execution you can see the Spark jobs processing for each cell?

[kyrrewk]

Yes, exactly that: the progress information about the job(s) and their stages that the query triggered.  We really need this information to be displayed in the application.

[Walter_C]

I would suggest you to submit this as a feature request through https://docs.databricks.com/en/resources/ideas.html#ideas