Trying to run databricks academy labs, but execution fails due to method to clearcache not whilelist

[Brammer88]

Hi there,

Im trying to run DE 2.1 - Querying Files Directly on my workspace with a default cluster configuration for found below,

but I cannot seem to run this file (or any other labs) as it gives me this error message 

Resetting the learning environment:

The execution of this command did not finish successfully

Python interpreter will be restarted. Python interpreter will be restarted.

Resetting the learning environment:

py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

Any pointers as to how i can fix this?Where do i need to go to whitelist this class and how would i do this? 

Thanks,

Bram

[Brammer88]

Hi Kaniz,

Im not sure why, but it seems like its not working after all and I was cheering to soon.

I have added a line 

spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache

to my cluster in the spark config (under advanced options), but i still get the same response;

"The execution of this command did not finish successfully

Python interpreter will be restarted. Python interpreter will be restarted.

Resetting the learning environment:

py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

Command skipped

Command skipped" 

Do you have any ideas as to what I am doing wrong?

Thanks,

Bram

[Brammer88]

Hi @Retired_mod,

Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way. 

Best,

Bram

[Brammer88]

works, thanks for the quick response! 

[Brammer88]

Hi @Kaniz,

Sorry, I was too fast with this reply, im not sure why, but it seems like its not working after all and I was cheering to soon.

I have added a line 

spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache

to my cluster in the spark config (under advanced options), but i still get the same response;

"The execution of this command did not finish successfully

Python interpreter will be restarted. Python interpreter will be restarted.

Resetting the learning environment:

py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

Command skipped

Command skipped" 

Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way. 

Best,

Bram

[Brammer88]

Hi @Retired_mod and databricks team,

Did you already found some other solution for this? 

Thanks,

Bram