topic Re: Orphan (?) files on Databricks S3 bucket in Data Engineering

Orphan (?) files on Databricks S3 bucket

alejandrofm — Tue, 04 Oct 2022 18:27:19 GMT

Hi, I'm seeing a lot of empty (and not) directories on routes like:

xxxxxx.jobs/FileStore/job-actionstats/

xxxxxx.jobs/FileStore/job-result/

xxxxxx.jobs/command-results/

Can I create a lifecycle to delete old objects (files/directories)? how many days? what is the best practice for this case?

Are there other directories that need a lifecycle configuration?

Thanks!

Re: Orphan (?) files on Databricks S3 bucket

Hubert-Dudek — Thu, 13 Oct 2022 15:16:40 GMT

In the Admin console, there are options to clean storage that you can use.

For everything configurable (DB locations, checkpoints), please use your storage control to control it.

Re: Orphan (?) files on Databricks S3 bucket

alejandrofm — Thu, 13 Oct 2022 21:51:01 GMT

Hi! I didn't know that, Purging right now, is there a way to schedule that so logs are retained for less time? Maybe I want to maintain the last 7 days for everything?

Thanks!

Re: Orphan (?) files on Databricks S3 bucket

Hubert-Dudek — Tue, 18 Oct 2022 14:10:23 GMT

During office hours, I asked one year ago to add purge cluster logs to API, and it was not even considered. I thought to set selenium to do that.

You can limit logging for the cluster by playing with log4j, for example, to put something like .sh script below on dbfs as the start script for the cluster (you need additionally specify log properties to be adjusted for drivers and executors):

#!/bin/bash
echo "Executing on Driver: $DB_IS_DRIVER"
if [[ $DB_IS_DRIVER = "TRUE" ]]; then
LOG4J_PATH="/home/ubuntu/databricks/spark/dbconf/log4j/driver/log4j.properties"
else
LOG4J_PATH="/home/ubuntu/databricks/spark/dbconf/log4j/executor/log4j.properties"
fi
echo "Adjusting log4j.properties here: ${LOG4J_PATH}"
echo "log4j.<custom-prop>=<value>" >> ${LOG4J_PATH}

In the notebook, you can disable logging by using:

sc.setLogLevel("OFF");

Additionally, for cluster config, you can set for delta files:

spark.databricks.delta.logRetentionDuration 3 days
spark.databricks.delta.deletedFileRetentionDuration 3 days

Re: Orphan (?) files on Databricks S3 bucket

alejandrofm — Sat, 29 Oct 2022 03:38:26 GMT

Not the best solution, will not implement something like this on prod, but it's the best answer, thanks!