โ04-17-2024 01:04 AM
Hi there,
Im trying to run DE 2.1 - Querying Files Directly on my workspace with a default cluster configuration for found below,
but I cannot seem to run this file (or any other labs) as it gives me this error message
โ04-17-2024 03:25 AM
Hi @Brammer88,
spark.databricks.pyspark.enablePy4JSecurity
to false
. However, this option is not recommended due to security implications.spark.jvm.class.allowlist
configuration property in your Spark configuration.org.apache.spark.sql.internal.CatalogImpl.clearCache
to the allowlist.โ04-18-2024 03:36 AM
Hi Kaniz,
Im not sure why, but it seems like its not working after all and I was cheering to soon.
I have added a line
spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache
to my cluster in the spark config (under advanced options), but i still get the same response;
a month ago
Hi @Kaniz,
Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way.
Best,
Bram
โ04-17-2024 11:41 PM
works, thanks for the quick response!
a month ago
Hi @Kaniz,
Sorry, I was too fast with this reply, im not sure why, but it seems like its not working after all and I was cheering to soon.
I have added a line
spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache
to my cluster in the spark config (under advanced options), but i still get the same response;
Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way.
Best,
Bram
3 weeks ago