Logs not coming up in the UI - while being written to DBFS

[FurqanAmin]

I have a few spark-submit jobs that are being run via Databricks workflows. I have configured logging in DBFS and specified a location in my GCS bucket.

The logs are present in that GCS bucket for the latest run but whenever I try to view them from the UI, it fails and doesn't return anything.

[Lakshay]

Are you able to see the logs by using the default cluster settings?

[FurqanAmin]

 By the default setting, do you mean that I enter the destination as None? instead of defining a DBFS location?

I tried using this too, but still I can't see the logs for that job

[Lakshay]

Yes, I meant to set it to None. Is the issue specific to any particular cluster? Or do you see the issue with all the clusters in your workspace?

[FurqanAmin]

So firstly, each task has a separate cluster when it comes to spark-submit jobs, so for those jobs, it is the same behavior across all jobs in the dev workspace that we have.

in the prod workspace, this issue is there for some tasks and not there for other tasks

[Lakshay]

It is a strange behavior. It needs a closer look. Could you please raise a support ticket?