topic Re: Library Management via Custom Compute Policies and ADF Job Triggering in Community Articles

Library Management via Custom Compute Policies and ADF Job Triggering

SashankKotta — Sun, 16 Jun 2024 07:35:28 GMT

This guide is intended for those looking to install libraries on a cluster using a Custom Compute Policy and trigger Databricks jobs from an Azure Data Factory (ADF) linked service. While many users rely on init scripts for library installation, it is recommended to use Custom Compute Policies for this purpose. Custom Compute Policies provide better control and management over library installations and can be configured to ensure compliance with organizational standards.

Follow these steps to achieve this:

Create a Custom Compute Policy:
- Navigate to the Policies tab: Compute > Policies tab.
- Create a policy, Navigate to the library and add libraries as per your requirement.
- Please note, you can install libraries using jar/wheel/Maven co-ordinates.
- For more details, follow the detailed instructions in the link: Create and manage compute policies.
- After creating the custom compute policy, you will need to use the policy ID when executing the notebook via Azure Data Factory (ADF).
Trigger a Databricks Notebook from ADF:
- Use the Custom Compute Policy created in Step 1 to initiate your Databricks notebook via Azure Data Factory (ADF).
- Using the ADF Linked service, we can trigger a notebook with our custom policy attached to the job/all-purpose cluster. Navigate to the Advanced option in the linked service and update the policy as shown below.

Summary:

This guide explains how to install libraries on a Databricks cluster using a Custom Compute Policy and trigger Databricks jobs from an Azure Data Factory (ADF) linked service. It details creating a Custom Compute Policy, noting the policy ID, and using this policy ID to execute notebooks via ADF. This method is recommended over using init scripts for library installation.

Re: Library Management via Custom Compute Policies and ADF Job Triggering

Sujitha — Tue, 25 Jun 2024 09:59:27 GMT

@SashankKotta Thank you for sharing!

Re: Library Management via Custom Compute Policies and ADF Job Triggering

hassan2 — Wed, 07 Aug 2024 14:46:11 GMT

The policyId doesn't seem to be working when using databricks pool instance. I am getting the following error

Cluster validation error: Validation failed for instance_pool_id, the value cannot be present (is {pool_id}); Validation failed for driver_instance_pool_id, the value cannot be present (is {pool_id}); Validation failed for azure_attributes.spot_bid_max_price from pool, the value must be present

Re: Library Management via Custom Compute Policies and ADF Job Triggering

SashankKotta — Wed, 07 Aug 2024 16:39:32 GMT

That is because your compute policy has all these issues:
1. instance_pool_id might have been set as forbidden please modify that, You need to pass the instance pool id in the compute policy you are using.
2. For driver instance pool id, please pass the driver instance pool id in your compute policy.
3. Please pass the correct value for your spot_bid_max_price in your compute policy.

For instance pools, I would suggest create a separate compute policy and use it.

Re: Library Management via Custom Compute Policies and ADF Job Triggering

hassan2 — Thu, 15 Aug 2024 16:56:14 GMT

Thanks, @SashankKotta for pointing this out. I was able to resolve the instance pool ID issue but spot_bid_max_price is still failing I am getting the following error

Cluster validation error: Validation failed for azure_attributes.spot_bid_max_price from pool, the value must be present

here is the policy definition in cluster policy

"azure_attributes.spot_bid_max_price": {
    "type": "unlimited",
    "defaultValue": 100,
    "hidden": false
},

Re: Library Management via Custom Compute Policies and ADF Job Triggering

SashankKotta — Wed, 21 Aug 2024 10:17:00 GMT

It seems, it is expecting a "value" parameter in your policy something like this.

"azure_attributes.spot_bid_max_price": {
"type": "unlimited","value": 100,"defaultValue": 100,"hidden": false
},

If it still fails try to remove the defaultValue parameter in your policy.

Re: Library Management via Custom Compute Policies and ADF Job Triggering

Wojciech_BUK — Mon, 14 Oct 2024 11:31:45 GMT

Hi @hassan2
I had same issue and found solution.
When I created POOL i created it as On-demand (not spot) and then policy only worked when I removed entire section "azure_attributes.spot_bid_max_price" from policy.
Looks like "azure_attributes.spot_bid_max_price" only works when you create POOL with SPOT instances only.

Re: Library Management via Custom Compute Policies and ADF Job Triggering

n1399 — Sun, 09 Mar 2025 04:54:28 GMT

How did you removed the inherited definitions ?

Re: Library Management via Custom Compute Policies and ADF Job Triggering

n1399 — Sun, 09 Mar 2025 13:07:19 GMT

I tried this way.

In the Compute Policy, instead of using the default Job Compute (since it comes with inherited definitions that cannot be removed, although they can be overridden), I cloned the definitions from Job Compute and created a custom policy. In this custom policy, I removed the 'spot_bid_max_price', and it worked out for me.