Databricks

FurqanAmin · 01-29-2024

We have a lot of jobs with spark-submit tasks, previously we were able to see the logs for the jobs. Now we are not able to see the logs in the DBX UI.We created a test job for this 'test_job_2' in our workspace to test it out. When the job finishes ...

FurqanAmin · 01-22-2024

I have a few spark-submit jobs that are being run via Databricks workflows. I have configured logging in DBFS and specified a location in my GCS bucket.The logs are present in that GCS bucket for the latest run but whenever I try to view them from th...

FurqanAmin · 01-26-2024

So firstly, each task has a separate cluster when it comes to spark-submit jobs, so for those jobs, it is the same behavior across all jobs in the dev workspace that we have.in the prod workspace, this issue is there for some tasks and not there for ...

FurqanAmin · 01-25-2024

By the default setting, do you mean that I enter the destination as None? instead of defining a DBFS location?I tried using this too, but still I can't see the logs for that job

FurqanAmin · 10-30-2023

@DmytroSokhach I think it works if you change mongo to mongodb in the options. and use spark.mongodb.read.connection.uri instead of spark.mongodb.input.uri as @silvadev suggested.

Databricks

User Stats

User Activity

Spark Logs inaccessible - from the UI and dbfs (GCS)

Logs not coming up in the UI - while being written to DBFS

Re: Logs not coming up in the UI - while being written to DBFS

Re: Logs not coming up in the UI - while being written to DBFS

Re: Spark connector to mongodb - mongo-spark-connector_2.12:10.1.1