topic asset bundles and compute policies in Data Engineering

asset bundles and compute policies

-werners- — Tue, 08 Oct 2024 13:32:31 GMT

Did anyone succeed in using already existing compute policies (created using the UI) in asset bundles for creating a job?
I defined the policy_id in the resources/job yml for the job_cluster, but when deploying I get errors saying spark version is not defined (this is defined in the policy), or other missing parameters (all defined in the policy).
So it seems that the policy is not fetched or applied.

Re: asset bundles and compute policies

szymon_dybczak — Tue, 08 Oct 2024 13:52:55 GMT

Hi @-werners- ,

I think you ran into the same kind of issue as the others in below discussion. There is some ongoing issue with TF provider, you can take a look at github thread:

DAB deployment fails with `Error: cannot create job: NumWorkers could be 0 only for SingleNode clusters` · Issue #1546 · databricks/cli · GitHub

Re: asset bundles and compute policies

-werners- — Tue, 08 Oct 2024 14:06:32 GMT

It looks like it, but I also get errors on non singlenode clusters.
But there might be an underlying issue with policy settings not being applied.
Tnx for the link though.

Re: asset bundles and compute policies

-werners- — Wed, 09 Oct 2024 07:04:14 GMT

So I figured it out.
You can actually refer existing cluster policies, but I made the mistake thinking all cluster config was added automatically by doing that.
In fact you still have to add some cluster config in the resources yaml:

- spark_version
- spark_conf + custom_tags (for singlenode clusters, see link Szymon posted)
- node_type_id + driver_type_id
When adding those in the yaml, deployment was possible.

I don't know why it works like it does, perhaps it is linked to the policy definition (f.e. optional attributes in the policy), but it would be nice if there was documentation on the requirements here.