I'm already using this "fix", but this goes against good development practices because you are hardcoding a filepath in your code. This filepath should be provided via a parameter, this is the reason that in most solutions ENVIRONMENT VARIABLES are used for , because the path might change at deployment time.

And as I mentioned before, following the Databricks documentation, you should be able to set environment variables using the spark_env_vars section. Is there anything wrong with my initial approach?