Hi! I'm using the code from here to execute a query on Databricks, it goes flawlessly, can follow it from the Spark UI, etc. The problem here is at the moment it seems the download of the result (spark is idle, there is a green check in the query history window, not showing total time and I don't have the data locally).

The issue here is that I don't know how to improve this, nor how much data is downloading, what percentage of it I got, what's remaining, etc. Even I don't find Ganglia to look at the network performance to understand what is happening.

Any idea of how to improve the download rate (or how to diagnose a little to improve it)

Another thing is that the endpoint is running and charging me while this happens, so thinking in download the result to S3 and getting it from there...but want to solve the monitor issue so I can understand how to improve this.

Thanks!