- 1868 Views
- 1 replies
- 0 kudos
How to Configure PySpark Jobs Using PEX
IssueI am attempting to create a PySpark job via the Databricks UI (with spark-submit) using the parameters below (dependencies are on the PEX file), but I am getting the an exception that the pex file does not exist. It's my understanding that the -...
- 1868 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi,I'm facing the same issue trying to execute a pyspark job with spark-submit.I have explored the same solution as you : --files optionspark.pyspark.driver.pythonspark.executorEnv.PEX_ROOTDo you make some progress in the resolution of the problem ?
- 0 kudos
- 1481 Views
- 0 replies
- 0 kudos
PySpark job with joins & write parquet operation fails with FetchFailedException
I'm working on a task where I transform a dataset and re-save it to an S3 bucket. This involves joining the dataset to two others, dropping fields from the initial dataset which overlapped with fields from the other two, hashing certain fields with p...
- 1481 Views
- 0 replies
- 0 kudos