topic Re: Asset Bundles cannot run job with single node job cluster in Data Engineering

Asset Bundles cannot run job with single node job cluster

Volker — Fri, 07 Jun 2024 14:02:19 GMT

Hello community,

we are deploying a job using asset bundles and the job should run on a single node job cluster. Here is the DAB job definition:

resources: jobs: example_job: name: example_job tasks: - task_key: main_task job_cluster_key: ${var.job_cluster_prefix} python_wheel_task: package_name: example_package entry_point: entrypoint named_parameters: config-path: "/Workspace${workspace.file_path}/config/app.conf" environment: "${bundle.target}" libraries: - whl: ./dist/*.whl job_clusters: - job_cluster_key: ${var.job_cluster_prefix} new_cluster: spark_version: 13.3.x-scala2.12 node_type_id: m4.2xlarge num_workers: 0 aws_attributes: first_on_demand: 0 availability: SPOT_WITH_FALLBACK zone_id: auto spot_bid_price_percent: 100 ebs_volume_type: GENERAL_PURPOSE_SSD ebs_volume_count: 1 ebs_volume_size: 100 tags: costs/environment: ${bundle.target} costs/stage: ${var.costs_stage} service: ${var.service} domain: ${var.domain} owner: ${var.domain}

Since yesterday we are facing the problem that the cluster spins up but does not run the code. Instead the following warning is printed to the Log4J output: "WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources"

The problem does not occurr when adding 1 worker node or when I edit the job cluster via the UI. So I guess, there is an issue with the asset bundles.

I checked the releases of the CLI and for the newest release "v.0.221.1" it states in the release notes: "This releases fixes an issue introduced in v0.221.0 where managing jobs with a single-node cluster would fail."

Also strangely: Locally I have version "v.0.218.0" of the CLI installed and when I deploy the job locally the code runs until some intended exception but instead of failing the job, the job keeps on running and the same message: "WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources" gets written to the Log4J output.

Did anybody else experience this issue and knows how to solve that?

Re: Asset Bundles cannot run job with single node job cluster

kunalmishra9 — Wed, 07 Aug 2024 08:14:59 GMT

Following. Having a similar issue in which setting num_workers to 0 doesn't work. When I deploy the bundle:

Error: cannot update job: NumWorkers could be 0 only for SingleNode clusters.

- job_cluster_key: ${bundle.name}

new_cluster:

cluster_name: ""

spark_version: 15.4.x-scala2.12

azure_attributes:

first_on_demand: 1

availability: ON_DEMAND_AZURE

spot_bid_max_price: -1

node_type_id: Standard_E32d_v4 # Worker Node: 32 cores & 256GB RAM

# driver_node_type_id: Standard_E

spark_env_vars:

PYSPARK_PYTHON: /databricks/python3/bin/python3

enable_elastic_disk: true

policy_id: ${var.ds_compute_policy_id}

data_security_mode: SINGLE_USER

runtime_engine: PHOTON

num_workers: 0

# autoscale:

# min_workers: 1

# max_workers: 2

Re: Asset Bundles cannot run job with single node job cluster

AlbertWang — Mon, 09 Sep 2024 01:48:37 GMT

@Volker @kunalmishra9 As answered by Ganesh Chandrasekaran , the below solution works. I tested. Btw, you need to keep both spark_conf and custom_tags.

job_clusters:

- job_cluster_key: job_cluster

new_cluster:

spark_version: 15.4.x-scala2.12

node_type_id: Standard_D4ads_v5

num_workers: 0

spark_conf:

"spark.databricks.cluster.profile": "singleNode"

"spark.master": "local[*, 4]"

custom_tags:

"ResourceClass": "SingleNode"

Re: Asset Bundles cannot run job with single node job cluster

AlbertWang — Mon, 09 Sep 2024 02:27:04 GMT

@Volker @kunalmishra9 Tested the below element is not necessary.
num_workers: 0

Re: Asset Bundles cannot run job with single node job cluster

Volker — Fri, 11 Oct 2024 07:22:22 GMT

Sorry for the late reply, this helped, thank you!

Re: Asset Bundles cannot run job with single node job cluster

kunalmishra9 — Wed, 05 Nov 2025 23:14:39 GMT

In case this is now breaking for anyone (as it is for me), there's an update here to follow along with on how to define single node compute!

https://github.com/databricks/databricks-sdk-py/issues/881