Hi @Akuhei05 how are you?

For the first topic, you can create a cell on your notebook that gets the spark configuration for max memory every time it is ran, relating to your cluster attached. For this, please see below:

spark_memory = spark.sparkContext.getConf().get('spark.executor.memory')
print(spark_memory)

 For the second point, when you say "determine the amount of memory utilized during the job" is it related to the maximum used in total? per worker? is it the sum of it?

Best,

Alessandro