topic Re: Set default "spark.driver.maxResultSize" from the notebook in Machine Learning

Set default "spark.driver.maxResultSize" from the notebook

mhansinger — Fri, 25 Feb 2022 12:07:47 GMT

Hello,

I would like to set the default "spark.driver.maxResultSize" from the notebook on my cluster. I know I can do that in the cluster settings, but is there a way to set it by code?

I also know how to do it when I start a spark session, but in my case I directly load from the feature store and want to transform my pyspark data frame to pandas.

from databricks import feature_store
import pandas as pd
import pyspark.sql.functions as f
from os.path import join
 
fs = feature_store.FeatureStoreClient()
 
prediction_data = fs.read_table(name=NAME)
 
prediction_data_pd = prediction_data.toPandas()

Re: Set default "spark.driver.maxResultSize" from the notebook

mhansinger — Fri, 25 Feb 2022 14:09:40 GMT

Hi @Kaniz Fatma thanks for your reply.

Not sure if that helps. When I check after execution of your code with

spark.conf.get("spark.driver.maxResultSize")

I still get the default "spark.dirver.maxResultSize", instead of 4096.

Re: Set default "spark.driver.maxResultSize" from the notebook

Atanu — Thu, 03 Mar 2022 13:41:52 GMT

@Maximilian Hansinger may be you can follow this -

https://kb.databricks.com/jobs/job-fails-maxresultsize-exception.html

Re: Set default "spark.driver.maxResultSize" from the notebook

Anonymous — Tue, 08 Mar 2022 21:56:42 GMT

@Maximilian Hansinger - Would you let us know how it goes, please?

Re: Set default "spark.driver.maxResultSize" from the notebook

Anonymous — Thu, 28 Apr 2022 16:37:53 GMT

Hi @Maximilian Hansinger

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark the answer as best? If not, please tell us so we can help you.

Thanks!