Spark Logs inaccessible - from the UI and dbfs (GCS)

FurqanAmin — Mon, 29 Jan 2024 08:43:35 GMT

We have a lot of jobs with spark-submit tasks, previously we were able to see the logs for the jobs. Now we are not able to see the logs in the DBX UI.
We created a test job for this 'test_job_2' in our workspace to test it out. When the job finishes executing, the logs for it are not accessible in the UI, even if we define a dbfs path of the Logging in the config of the job, the logs are still not accessible on the UI nor the dbfs location.
While the job is running, the logs are accessible, but as soon as it terminates the logs disappear.

Re: Spark Logs inaccessible - from the UI and dbfs (GCS)

Yeshwanth — Wed, 31 Jan 2024 15:55:13 GMT

@FurqanAmin Could you please attach a screenshot of this?

topic Re: Spark Logs inaccessible - from the UI and dbfs (GCS) in Data Engineering

Spark Logs inaccessible - from the UI and dbfs (GCS)

Re: Spark Logs inaccessible - from the UI and dbfs (GCS)