I am trying to run a multi file python job in databricks without using notebooks. I have tried setting this up by:

• creating a docker image using the DBRT 10.4 LTS as a base and adding the zipped python application to that.
• make a call to the run submit endpoint with this payload:

{
    "tasks": {
        "task_key": "test-run-8",
        "spark_submit_task": {
            "parameters": [
                "--py-files",
                "/app.zip",
                "/app.zip/__main__.py"
            ]
        },
        "new_cluster": {
            "num_workers": 1,
            "spark_version": "11.1.x-scala2.12",
            "aws_attributes": {
                "first_on_demand": 1,
                "availability": "SPOT_WITH_FALLBACK",
                "zone_id": "us-west-2a",
                "instance_profile_arn": "<instance profile ARN>",
                "spot_bid_price_percent": 100,
                "ebs_volume_count": 0
            },
            "node_type_id": "i3.xlarge",
            "docker_image": {
                "url": "<aws-account-number>.dkr.ecr.us-west-2.amazonaws.com/spark-app:0.1.17"
            }
        }
    }
}

The application tries to read a JSON file and load it into a new delta lake table. Unfortunately this does not work as intended. Here is what I have found:

• When I run the code in the application out of a notebook, it works normally
• when running using the the jobs endpoint I don't see the table at all in the databricks UI
• When checking the S3 bucket I do see a folder created for the database and some parquet files of the table.
• running any queries on this table fails with a not found error.
• when checking the driver logs I see the following:

...
22/08/19 02:38:28 WARN DefaultTableOwnerAclClient: failed to update the table owner when create/drop table.
java.lang.IllegalStateException: Driver context not found
...

some extra considerations:

• I'd like to be able to use docker images for our deployment if possible since it matches our current CI/CD pattern
• Failing that I'd be ok with a spark_python_task, but I have not been able to get this to work when I have multiple python files
• I want to avoid using notebooks for deploying applications.

Any help with understanding and fixing this error would be very appreciated.

Regards,

Umar