When you say "Databricks Job", do you mean a custom SQL code you created in an stored procedure or function? Saying this because if you're really trying to run a real Databricks Job (lakeflow), it would be much better to establish another strategy different from that initial one. I mean, create firstly the Databricks Job and then, add type of tasks better for you scenario (https://docs.databricks.com/aws/en/jobs/configure-task#types-of-tasks )