Output is usually something related to print() collect() etc

In the documentation which you mentioned is spark config command to remove totally stdout (spark.databricks.driver.disableScalaOutput true). I know that it is not what you want to use but maybe it could be helpful to diagnose is a problem with logs or script output.

Not many people use spark_python_task rather all use notebooks (eventually together with files in repos or wheel) so maybe someone from inside databricks would need to help.