Databricks Community

Rahul2025 · 02-02-2023

Hi,We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (clust...

Rahul2025 · 02-02-2023

Hi,We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (clust...

Rahul2025 · 02-21-2023

yes @Sundar Raman, the location is not a DBFS mount. We tried this in a workspace where DBR 10.4LTS and 11.3LTS are being used.

Rahul2025 · 02-15-2023

Yes @Sundar Raman, but that also gives the same error.

Rahul2025 · 02-14-2023

Thank you @Sundar Raman for your response. Yes, we referred the same link that you shared and we currently use the Cluster-scoped script (following option #4) as against Global (following option #3) because the script needs to be executed only for o...

Rahul2025 · 02-13-2023

Thank you for your response. One of the other options is to put them in the following file from init script./databricks/spark/conf/

Rahul2025 · 02-13-2023

Thank you @karthik p for your response. I'm trying to upload a script named db_init.sh (of 10kb size) to /databricks/init folder and getting the following error -Error occurred when processing file db_init.sh: Server responded with 0 code.Had it be...

Databricks Community

User Stats

User Activity

Limitation on size of init script

Make environment variables defined in init script available to Spark JVM job?

Re: Limitation on size of init script

Re: Limitation on size of init script

Re: Limitation on size of init script

Re: Make environment variables defined in init script available to Spark JVM job?

Re: Limitation on size of init script