Trying to run databricks academy labs, but execution fails due to method to clearcache not whilelist

Brammer88
New Contributor III

‎04-17-2024 01:04 AM

Hi there,

Im trying to run DE 2.1 - Querying Files Directly on my workspace with a default cluster configuration for found below,

Brammer88_0-1713340930496.png

but I cannot seem to run this file (or any other labs) as it gives me this error message 

 

Resetting the learning environment:
The execution of this command did not finish successfully
 
Python interpreter will be restarted. Python interpreter will be restarted.
 
 
Resetting the learning environment:
 
py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl
 
Any pointers as to how i can fix this?Where do i need to go to whitelist this class and how would i do this? 
 
Thanks,
Bram

 

 

0 Kudos

Brammer88
New Contributor III
In response to Retired_mod

‎04-18-2024 03:36 AM

Hi Kaniz,

Im not sure why, but it seems like its not working after all and I was cheering to soon.

I have added a line 

spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache

to my cluster in the spark config (under advanced options), but i still get the same response;

"The execution of this command did not finish successfully
 
Python interpreter will be restarted. Python interpreter will be restarted.
 
 
Resetting the learning environment:
py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl
Command skipped
Command skipped" 
 
Do you have any ideas as to what I am doing wrong?
 
Thanks,
Bram

 

0 Kudos

Brammer88
New Contributor III
In response to Brammer88

‎04-22-2024 12:25 AM

Hi @Retired_mod,

Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way. 

Best,

Bram

 

Brammer88
New Contributor III

‎04-17-2024 11:41 PM

works, thanks for the quick response! 

Brammer88
New Contributor III
In response to Brammer88

‎04-22-2024 01:43 AM

Hi @Kaniz,

Sorry, I was too fast with this reply, im not sure why, but it seems like its not working after all and I was cheering to soon.

I have added a line 

spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache

to my cluster in the spark config (under advanced options), but i still get the same response;

"The execution of this command did not finish successfully
 
Python interpreter will be restarted. Python interpreter will be restarted.
 
 
Resetting the learning environment:
py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl
Command skipped
Command skipped" 
 

Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way. 

Best,

Bram

0 Kudos

Brammer88
New Contributor III

‎04-30-2024 01:34 PM

Hi @Retired_mod and databricks team,

Did you already found some other solution for this? 

Thanks,

Bram

0 Kudos