Databricks Community

Jaron · 06-19-2024

Recently, when I was using databricks for deep learning I ran into an issue, i.e., after a certain amount of time of execution the cluster would break and restart. The logs are as below:The Event logs displayed "Metastore is down; Driver is up but is...

Jaron · 06-29-2024

Hi, Walter_C, I have tried to redirect the .log file to other destination. However, I found that redirection through create cluster→Advanced Options→Logging→Destination is a copy rather than a move. This means that the driver log will still increase....

Jaron · 06-22-2024

Thanks for your reply. However, although I modify the create cluster→Advanced Options→Logging→Destination to a destination, the "echo: write error: no space left on device" still appears. I change the destination to "/dbfs/FileStore", where the space...

Jaron · 06-18-2024

Thanks! But I'm running the python script file via workflow→jobs, so I can't submit "spark.catalog.clearCache()" via notebooks because they're isolated. Is there any way out of this situation?.For another question, may I ask if the ''memory'' you me...

Jaron · 06-17-2024

Hi, I meet the seme problem when I train a DeepLearning model. Could you tell me where to set this 'spark.catalog.clearCache()'? Thanks!

Databricks Community

User Stats

User Activity

Where the "Driver logs" are stored by default, and How much default space to store it.

Re: Where the "Driver logs" are stored by default, and How much default space to store it

Re: Where the "Driver logs" are stored by default, and How much default space to store it

Re: Driver is up but is not responsive, likely due to GC.

Re: Driver is up but is not responsive, likely due to GC.