Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-10-2025 10:15 AM
@Alberto_Umana In addition to my last comment:
For adjusting the spark.sql.shuffle.partitions and spark.executor.memory, I tried this but I was still seeing the same error
spark = (
SparkSession.builder
.appName("GA4 Bronze Table Ingestion")
.config("spark.sql.shuffle.partitions", "100")
.config("spark.executor.memory", "4g")
.config("spark.driver.memory", "4g")
.config("spark.sql.execution.arrow.pyspark.enabled", "true")
.getOrCreate()
)
