Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
We have a lot of jobs with spark-submit tasks, previously we were able to see the logs for the jobs. Now we are not able to see the logs in the DBX UI. We created a test job for this 'test_job_2' in our workspace to test it out. When the job finishes executing, the logs for it are not accessible in the UI, even if we define a dbfs path of the Logging in the config of the job, the logs are still not accessible on the UI nor the dbfs location. While the job is running, the logs are accessible, but as soon as it terminates the logs disappear.
