Current Cluster Config:
Standard_DS3_v2 (14GB, 4 Cores) 2-6 workers
Standard_DS3_v2 (14GB, 4Cores) for driver
Runtime: 10.4x-scala2.12
We want to overwrite a temporary delta table with new records. The records will be load by another delta table and transformed in a notebook. This records have a column with a large string value.
We think the error occours because the driver have to handle to many memory. So we tested different configurations with the cluster (e.g. spark.executor.memory, spark.driver.memory, ...)
We also tested repartitioning and maxRowsInMemory. Sometimes our job runs, but at the most time we get such errors.
e.g.
Notebook-Error:
o1161.saveAsTable, o1158.saveAsTable, ...
193
194 print(f"Overwrites records in delta table '{full_table_name}'.")
--> 195 df.write.format("delta")\
196 .partitionBy("Year", "TypeOfTapChanger") \
197 .mode("overwrite") \
/databricks/spark/python/pyspark/sql/readwriter.py in saveAsTable(self, name, format, mode, partitionBy, **options)
804 if format is not None:
805 self.format(format)
--> 806 self._jwrite.saveAsTable(name)
807
808 def json(self, path, mode=None, compression=None, dateFormat=None, timestampFormat=None,
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
1302
1303 answer = self.gateway_client.send_command(command)
-> 1304 return_value = get_return_value(
1305 answer, self.gateway_client, self.target_id, self.name)
1306
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
115 def deco(*a, **kw):
116 try:
--> 117 return f(*a, **kw)
118 except py4j.protocol.Py4JJavaError as e:
119 converted = convert_exception(e.java_exception)
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o1161.saveAsTable.
: org.apache.spark.SparkException: Job aborted.
...
...
Driver-Error:
23/02/15 10:58:38 INFO Executor: Finished task 2.0 in stage 139.0 (TID 926). 2578 bytes result sent to driver
23/02/15 10:58:38 INFO CoarseGrainedExecutorBackend: Got assigned task 955
23/02/15 10:58:38 INFO TorrentBroadcast: Started reading broadcast variable 144 with 1 pieces (estimated total size 4.0 MiB)
23/02/15 10:58:39 INFO MemoryStore: Block broadcast_144_piece0 stored as bytes in memory (estimated size 42.8 KiB, free 3.0 GiB)
23/02/15 10:58:39 INFO TorrentBroadcast: Reading broadcast variable 144 took 1077 ms
23/02/15 10:58:39 INFO MemoryStore: Block broadcast_144 stored as values in memory (estimated size 149.3 KiB, free 3.0 GiB)
23/02/15 10:58:39 INFO Executor: Running task 7.0 in stage 140.0 (TID 955)
23/02/15 10:58:39 INFO CodeGenerator: Code generated in 200.255017 ms
23/02/15 10:58:40 INFO TorrentBroadcast: Started reading broadcast variable 141 with 1 pieces (estimated total size 4.0 MiB)
23/02/15 10:58:41 INFO CodeGenerator: Code generated in 1729.166417 ms
23/02/15 10:58:46 ERROR Utils: Uncaught exception in thread stdout writer for /databricks/python/bin/python
java.lang.OutOfMemoryError: Java heap space
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:78)
at org.apache.spark.sql.execution.python.PythonUDFRunner$$anon$2.read(PythonUDFRunner.scala:68)
...
...
Executors in UI:
We hope you can help us, to understand and fix this problem.
