topic Cannot create jobs with jobs api - Azure databricks - private network in Data Engineering

Cannot create jobs with jobs api - Azure databricks - private network

SyedGhouri — Wed, 20 Mar 2024 23:29:10 GMT

Hi

I'm trying to deploy the databricks jobs from dev to prod environment. I have jobs in dev environment and using azure devops, I deployed the jobs in the code format to prod environment. Now when I use the post method to create the job programmatically, it always give 404 error. All the api calls give 404 error. It works fine in dev environment but not in prod. Our prod environment does not have access to the public network whereas the dev has (both of the environments are in different resource groups.)

So my understanding is since the prod is behind private network the api calls are not working? How do I enable the api calls to work in prod which does not have public access.

Any help is appreciated.

Thanks
Syed

Re: Cannot create jobs with jobs api - Azure databricks - private network

daniel_sahal — Thu, 21 Mar 2024 08:14:58 GMT

@SyedGhouri
You need to setup self-hosted Azure DevOps Agent inside your VNET.

Re: Cannot create jobs with jobs api - Azure databricks - private network

SyedGhouri — Thu, 21 Mar 2024 15:24:51 GMT

Hi @daniel_sahal

Thank you for your message.

I have a self-hosted Azure DevOps agent in VNET which I'm using to do the deployment from Azure DevOps. The deployment works fine. The code is deployed from dev to prod. Once the code is deployed, I have a file which creates databricks jobs using jobs API. Its this file which is not working. Im running this file from the databricks, not Azure DevOps.

Thanks
Syed