@Thijs van den Berg :
When creating a job in Databricks, you can reference code that is inside the container by using the dbutils module. Here's an example of how you could reference a Python file called myscript.py that is located in the /opt/myapp directory of the container:
import os
dbutils.fs.cp("file:/opt/myapp/myscript.py", "dbfs:/mnt/my-mount-point/myscript.py")
os.system("python /dbfs/mnt/my-mount-point/myscript.py arg1 arg2 arg3")
In this example, we first copy the myscript.py file from the container file system to a DBFS mount point using the dbutils.fs.cp() method. Then we run the Python script using the os.system() method and passing in any command line arguments. You can also use the databricks-cli to automate the creation of jobs and the upload of files to DBFS. Here's an example:
databricks fs cp /opt/myapp/myscript.py dbfs:/mnt/my-mount-point/myscript.py
databricks jobs create --name "My Job" --python-task "python /dbfs/mnt/my-mount-point/myscript.py arg1 arg2 arg3" --max-retries 0
This example uses the databricks-cli to copy the myscript.py file to DBFS and then creates a new job with a Python task that runs the script with command line arguments.
I hope this helps! Let me know if you have any further questions.