Databricks logging of SQL queries to DBFS

sparkplug
Contributor

Hi

Our costs has suddenly spiked due to logging of a lot of SQL query outputs to DBFS. We haven't made any changes to enable this. How can we disable this feature?

saurabh18cs
Honored Contributor III

# Check current Spark configurations
spark.conf.get("spark.sql.queryExecutionListeners", "")

# Disable query logging
spark.conf.set("spark.sql.queryExecutionListeners", "")

 

or

GO to your cluster -> EDIT -> ADVANCED OPTION -> logging tab and set to none

saurabh18cs_0-1737977809328.png

 

 

sparkplug
Contributor

I don't get any output when running the following, I have the destination set to dbfs . But it was only supposed to be for cluster logs and not for query execution outputs to be stored in DBFS. Any idea if this is expected behavior.

spark.conf.get("spark.sql.queryExecutionListeners", "")

saurabh18cs
Honored Contributor III

I would suggest to give a try by setting it to None and see if that helps. please share your findings.

Hi @saurabh18cs ,

I believe setting it to none would resolve this issue. 
Also can you please share all cluster configs using which you are running the SQL query?