topic Trying to run databricks academy labs, but execution fails due to method to clearcache not whilelist in Data Engineering

Trying to run databricks academy labs, but execution fails due to method to clearcache not whilelist

Brammer88 — Wed, 17 Apr 2024 08:04:18 GMT

Hi there,

Im trying to run DE 2.1 - Querying Files Directly on my workspace with a default cluster configuration for found below,

but I cannot seem to run this file (or any other labs) as it gives me this error message

Resetting the learning environment:

The execution of this command did not finish successfully

Python interpreter will be restarted. Python interpreter will be restarted.

Resetting the learning environment:

py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

Any pointers as to how i can fix this?Where do i need to go to whitelist this class and how would i do this?

Thanks,

Bram

Re: Trying to run databricks academy labs, but execution fails due to method to clearcache not while

Brammer88 — Thu, 18 Apr 2024 06:41:57 GMT

works, thanks for the quick response!

Re: Trying to run databricks academy labs, but execution fails due to method to clearcache not while

Brammer88 — Thu, 18 Apr 2024 10:36:08 GMT

Hi Kaniz,

Im not sure why, but it seems like its not working after all and I was cheering to soon.

I have added a line

spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache

to my cluster in the spark config (under advanced options), but i still get the same response;

"The execution of this command did not finish successfully

Python interpreter will be restarted. Python interpreter will be restarted.

Resetting the learning environment:

py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

Command skipped

Command skipped"

Do you have any ideas as to what I am doing wrong?

Thanks,

Bram

Re: Trying to run databricks academy labs, but execution fails due to method to clearcache not while

Brammer88 — Mon, 22 Apr 2024 07:25:29 GMT

Hi @Retired_mod,

Do you have any other solution/idea as to what might cause the issue? Or do you need any more details for investigation? Thanks again for your help! I am completely stuck as I am unable to do any work with the labs in this way.

Best,

Bram

Re: Trying to run databricks academy labs, but execution fails due to method to clearcache not while

Brammer88 — Mon, 22 Apr 2024 08:43:26 GMT

Hi @Kaniz,

Sorry, I was too fast with this reply, im not sure why, but it seems like its not working after all and I was cheering to soon.

I have added a line

spark.jvm.class.allowlist org.apache.spark.sql.internal.CatalogImpl.clearCache

to my cluster in the spark config (under advanced options), but i still get the same response;

"The execution of this command did not finish successfully

Python interpreter will be restarted. Python interpreter will be restarted.

Resetting the learning environment:

py4j.security.Py4JSecurityException: Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on class class org.apache.spark.sql.internal.CatalogImpl

Command skipped

Command skipped"

Best,

Bram

Re: Trying to run databricks academy labs, but execution fails due to method to clearcache not while

Brammer88 — Tue, 30 Apr 2024 20:34:19 GMT

Hi @Retired_mod and databricks team,

Did you already found some other solution for this?

Thanks,

Bram