I am trying to create a multi-task Databricks Job in Azure Cloud with its own cluster.
Although I was able to create a single task job without any issues, the code to deploy the multi-task job fails due to the following cluster validation error:
error: 1 error occurred:
* cannot create job: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size
he code to create the Job is the following:
job = Job(
resource_name = f"{job_name}-job",
args=JobArgs(
name = f"{job_name}-job",
job_clusters=[
JobJobClusterArgs(
job_cluster_key="pulumiTest-basic-cluster",
new_cluster=JobJobClusterNewClusterArgs(
spark_version="13.3.x-scala2.12",
cluster_name="",
num_workers=0,
node_type_id="Standard_DS3_v2",
enable_elastic_disk=True,
runtime_engine="STANDARD",
spark_conf={
# f"fs.azure.account.key.{self.storage_account_name}.dfs.core.windows.net": "{{secrets/pulumiTest-secret-scope/puluTest-storage-access-token}}"
"spark.master": "local[*,4]",
"spark.databricks.cluster.profile": "singleNode"
},
custom_tags={
"ResourceClass": "SingleNode"
},
data_security_mode="LEGACY_SINGLE_USER_STANDARD"
)
)
],
computes=[
JobComputeArgs(
compute_key="landing_task",
spec=JobComputeSpecArgs(kind="spark_python_task")
),
JobComputeArgs(
compute_key="staging_task",
spec=JobComputeSpecArgs(kind="spark_python_task")
),
JobComputeArgs(
compute_key="refined_task",
spec=JobComputeSpecArgs(kind="spark_python_task")
)
],
tasks = [
JobTaskArgs(
task_key="landing_task",
job_cluster_key="pulumiTest-basic-cluster",
spark_python_task=JobSparkPythonTaskArgs(
python_file="/pipelineExample/landing.py",
source="GIT"
),
run_if="ALL_SUCCESS",
libraries=[
JobLibraryArgs(
whl=whl_path
)
]
),
JobTaskArgs(
task_key="staging_task",
job_cluster_key="pulumiTest-basic-cluster",
spark_python_task=JobSparkPythonTaskArgs(
python_file="/pipelineExample/staging.py",
source="GIT"
),
depends_ons=[
JobTaskDependsOnArgs(
task_key="landing_task"
)
],
run_if="ALL_SUCCESS",
libraries=[
JobLibraryArgs(
whl=whl_path
)
]
),
JobTaskArgs(
task_key="refined_task",
job_cluster_key="pulumiTest-basic-cluster",
spark_python_task=JobSparkPythonTaskArgs(
python_file="/pipelineExample/refined.py",
source="GIT"
),
depends_ons=[
JobTaskDependsOnArgs(
task_key="staging_task"
)
],
run_if="ALL_SUCCESS",
libraries=[
JobLibraryArgs(
whl=whl_path
)
]
)
],
git_source=JobGitSourceArgs(
url=git_url,
provider="gitHub",
branch="main"
),
format="MULTI_TASK"
)
)
)
pulumi.export('Job URL', job.url)
Does anyone know where the problem could be?
