databricks-connect in Azure DevOps Pipeline jobs runs not showing on remote cluster

[DataEngineer92]

Hi Team,

I am trying to run Azure DevOps pipeline as mentioned in the blog below.

​https://benalexkeen.com/unit-testing-with-databricks-part-2/

The pipeline is running successfully however I am not able to see any runs in remote cluster.

Does databricks-connect only use as a part of CI process? And it is decoupled with Remote cluster?

Do I need to deploy the notebooks as a part of CD process and run the notebooks in order to show that under job runs?

I am new to databricks so please guide me

[Anonymous]

@Rey Jhon​ :

Databricks-connect is used to run notebooks locally on your development environment, while remote clusters are used to run notebooks on a distributed cluster environment. In the context of the blog you linked, databricks-connect is used for unit testing notebooks locally, and the remote cluster is used to run the notebooks in a distributed environment.

If you want to see the job runs on the remote cluster, you need to deploy the notebooks to the remote cluster using one of the available deployment methods such as the Databricks CLI or the Databricks API. Once the notebooks are deployed, you can create and run jobs in Databricks to execute the deployed notebooks on the remote cluster.

In summary, you can use databricks-connect to test and develop notebooks locally, and then deploy them to the remote cluster to run them in a distributed environment and view the job runs.

[DataEngineer92]

@Suteja Kanuri​ : The pipeline is suggesting to use databricks-connect configure and after that pytest cases running.

https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops

[Anonymous]

Hi @Rey Jhon​ 

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!