Databricks logging of SQL queries to DBFS

[sparkplug]

Hi

Our costs has suddenly spiked due to logging of a lot of SQL query outputs to DBFS. We haven't made any changes to enable this. How can we disable this feature?

[saurabh18cs]

# Check current Spark configurations
spark.conf.get("spark.sql.queryExecutionListeners", "")

# Disable query logging
spark.conf.set("spark.sql.queryExecutionListeners", "")

or

GO to your cluster -> EDIT -> ADVANCED OPTION -> logging tab and set to none

[sparkplug]

I don't get any output when running the following, I have the destination set to dbfs . But it was only supposed to be for cluster logs and not for query execution outputs to be stored in DBFS. Any idea if this is expected behavior.

spark.conf.get("spark.sql.queryExecutionListeners", "")

[saurabh18cs]

I would suggest to give a try by setting it to None and see if that helps. please share your findings.

[Sidhant07]

Hi @saurabh18cs ,

I believe setting it to none would resolve this issue. 
Also can you please share all cluster configs using which you are running the SQL query?