topic Failure to locate configuration file when using spark-submit task in Data Engineering

Failure to locate configuration file when using spark-submit task

Tham99 — Fri, 13 Sep 2024 09:39:16 GMT

Hello,

We are trying to run a job with a spark-sumit task on cluster mode, this spark submit task requires a configuration file application.conf that we provide using --files flag option in the spark-submit parameters and put an alias on it using \#application.conf (\ is just used to avoid highlight in this text box), we specify the configuration file using "spark.driver.extraJavaOptions=-Dconfig.file=application.conf and we are getting a java.io.FileNotFoundException each time, either this file is placed on DBFS or local file system. log4j logs show that file is fetched and copied with timestamp but then fails when main class tries to parse it. Any idea how to fix this issue?

Thanks in advance

Re: Failure to locate configuration file when using spark-submit task

VZLA — Wed, 11 Dec 2024 19:12:14 GMT

Hi @Tham99 thanks for your question!

If masking sensitive data is possible, could you please share the Driver log with the exception?

Re: Failure to locate configuration file when using spark-submit task

VZLA — Thu, 12 Dec 2024 17:32:35 GMT

@Tham99 would it be possible to share the java.io.FileNotFoundException stacktrace? And refernece from the Driver log about the file localization process.