topic Re: Logs not coming up in the UI - while being written to DBFS in Data Engineering

Logs not coming up in the UI - while being written to DBFS

FurqanAmin — Mon, 22 Jan 2024 11:17:12 GMT

I have a few spark-submit jobs that are being run via Databricks workflows. I have configured logging in DBFS and specified a location in my GCS bucket.

The logs are present in that GCS bucket for the latest run but whenever I try to view them from the UI, it fails and doesn't return anything.

Re: Logs not coming up in the UI - while being written to DBFS

Lakshay — Mon, 22 Jan 2024 15:08:17 GMT

Are you able to see the logs by using the default cluster settings?

Re: Logs not coming up in the UI - while being written to DBFS

FurqanAmin — Thu, 25 Jan 2024 13:48:49 GMT

By the default setting, do you mean that I enter the destination as None? instead of defining a DBFS location?

I tried using this too, but still I can't see the logs for that job

Re: Logs not coming up in the UI - while being written to DBFS

Lakshay — Thu, 25 Jan 2024 17:53:28 GMT

Yes, I meant to set it to None. Is the issue specific to any particular cluster? Or do you see the issue with all the clusters in your workspace?

Re: Logs not coming up in the UI - while being written to DBFS

FurqanAmin — Fri, 26 Jan 2024 10:18:33 GMT

So firstly, each task has a separate cluster when it comes to spark-submit jobs, so for those jobs, it is the same behavior across all jobs in the dev workspace that we have.

in the prod workspace, this issue is there for some tasks and not there for other tasks

Re: Logs not coming up in the UI - while being written to DBFS

Lakshay — Fri, 26 Jan 2024 12:43:59 GMT

It is a strange behavior. It needs a closer look. Could you please raise a support ticket?