topic Monitor progress when using databricks-connect in Data Engineering

Monitor progress when using databricks-connect

kyrrewk — Sat, 18 Jan 2025 09:52:44 GMT

When using databricks-connect how can you monitor the progress? Ideally, we want something similar to what you get in the Databricks notebook, i.e., information about the jobs/stages. We are using Python.

Re: Monitor progress when using databricks-connect

Walter_C — Sat, 18 Jan 2025 13:58:04 GMT

When you refer to progress, you mean that during the notebook execution you can see the Spark jobs processing for each cell?

Re: Monitor progress when using databricks-connect

kyrrewk — Sat, 18 Jan 2025 14:35:00 GMT

Yes, exactly that: the progress information about the job(s) and their stages that the query triggered. We really need this information to be displayed in the application.

Re: Monitor progress when using databricks-connect

Walter_C — Mon, 20 Jan 2025 14:04:08 GMT

I would suggest you to submit this as a feature request through https://docs.databricks.com/en/resources/ideas.html#ideas