topic Databricks logging of SQL queries to DBFS in Data Engineering

Databricks logging of SQL queries to DBFS

sparkplug — Mon, 27 Jan 2025 08:22:46 GMT

Hi

Our costs has suddenly spiked due to logging of a lot of SQL query outputs to DBFS. We haven't made any changes to enable this. How can we disable this feature?

Re: Databricks logging of SQL queries to DBFS

saurabh18cs — Mon, 27 Jan 2025 11:37:05 GMT

# Check current Spark configurations
spark.conf.get("spark.sql.queryExecutionListeners", "")

# Disable query logging
spark.conf.set("spark.sql.queryExecutionListeners", "")

GO to your cluster -> EDIT -> ADVANCED OPTION -> logging tab and set to none

Re: Databricks logging of SQL queries to DBFS

sparkplug — Mon, 27 Jan 2025 15:44:32 GMT

I don't get any output when running the following, I have the destination set to dbfs . But it was only supposed to be for cluster logs and not for query execution outputs to be stored in DBFS. Any idea if this is expected behavior.

spark.conf.get("spark.sql.queryExecutionListeners", "")

Re: Databricks logging of SQL queries to DBFS

saurabh18cs — Wed, 29 Jan 2025 13:44:20 GMT

I would suggest to give a try by setting it to None and see if that helps. please share your findings.

Re: Databricks logging of SQL queries to DBFS

Sidhant07 — Thu, 30 Jan 2025 07:49:34 GMT

Hi @saurabh18cs ,

I believe setting it to none would resolve this issue.
Also can you please share all cluster configs using which you are running the SQL query?