topic Re: Can you help with this error please? Issue when using a new high concurrency cluster in Data Engineering

Can you help with this error please? Issue when using a new high concurrency cluster

TJS — Fri, 08 Oct 2021 15:24:48 GMT

Hello, I am trying to use MLFlow on a new high concurrency cluster but I get the error below. Does anyone have any suggestions? It was working before on a standard cluster. Thanks.

py4j.security.Py4JSecurityException: Method public int org.apache.spark.SparkContext.maxNumConcurrentTasks() is not whitelisted on class class org.apache.spark.SparkContext

--------------------------------------------------------------------------- Py4JError Traceback (most recent call last) <command-2769834740329298> in <module> 32 # Greater parallelism will lead to speedups, but a less optimal hyperparameter sweep. 33 # A reasonable value for parallelism is the square root of max_evals. ---> 34 spark_trials = SparkTrials(parallelism=10) 35 36 /databricks/.python_edge_libs/hyperopt/spark.py in __init__(self, parallelism, timeout, loss_threshold, spark_session) 101 ) 102 # maxNumConcurrentTasks() is a package private API --> 103 max_num_concurrent_tasks = self._spark_context._jsc.sc().maxNumConcurrentTasks() 104 spark_default_parallelism = self._spark_context.defaultParallelism 105 self.parallelism = self._decide_parallelism( /databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args) 1303 answer = self.gateway_client.send_command(command) 1304 return_value = get_return_value( -> 1305 answer, self.gateway_client, self.target_id, self.name) 1306

Re: Can you help with this error please? Issue when using a new high concurrency cluster

Anonymous — Fri, 08 Oct 2021 20:44:15 GMT

Hello, @Tom Soto! My name is Piper and I'm a moderator for Databricks. It's great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else I will follow up shortly with a response.

Re: Can you help with this error please? Issue when using a new high concurrency cluster

jose_gonzalez — Fri, 08 Oct 2021 23:42:02 GMT

hi @Tom Soto ,

The error message is coming from your high concurrency cluster's security model. This is build-in security model to restrict access to your data. Your code might work on standard cluster but not on high concurrency clusters.

Re: Can you help with this error please? Issue when using a new high concurrency cluster

TJS — Wed, 13 Oct 2021 15:33:45 GMT

Thank you for your response. I appreciate this but are you aware of any work around to use a high concurrency cluster as it is a special databricks function that is the issue

Re: Can you help with this error please? Issue when using a new high concurrency cluster

Pradeep54 — Tue, 19 Oct 2021 13:05:56 GMT

@Tom Soto We have a workaround for this. This cluster spark configuration setting will disable py4jSecurity while still enabling passthrough

spark.databricks.pyspark.enablePy4JSecurity false

Re: Can you help with this error please? Issue when using a new high concurrency cluster

TJS — Fri, 29 Oct 2021 13:36:17 GMT

Thank you very much. This workaround worked for me.

Re: Can you help with this error please? Issue when using a new high concurrency cluster

Piper_Wilson — Fri, 29 Oct 2021 22:04:44 GMT

@Tom Soto - If Pradpalnis fully answered your question, would you be happy to mark their answer as best so that others can quickly find the solution?