Running unit tests and hyperopt causes a broadcast variable exception

[Boyan]

Hello,

We are using hyperopt to train a model with relatively large train dataset.

We've experience some performance issues and following the suggestions in this notebook, we broadcasted the dataset.

To verify that broadcasting the dataset resolved the performance issue, we did an experiment using Databricks Runtime for Machine Learning and a Notebook. We did see a significant performance boost.

To deploy our code, we package it as a .whl file and utilize python jobs to deploy it to an Azure Databricks Service. Provided we run the job using Databricks Runtime for Machine Learning, we do not have any issues.

We run into the following issues "Broadcast variable '5' not loaded!", when we run unit tests for our jobs locally or via our CICD pipelines.

This appears to be a known bug in the hyperopt library and there is a fix merged to master but it is not released.

Databricks Runtime for Machine Learning ships with a Databricks fork of hyperopt - version 0.2.7+db1, which has a fix too.

Given that this fork is only available on Databricks Runtimes for Machine Learning, what is the recommended approach to run unit tests on CI/CD infrastructure or local development machines?

[Kaniz]

Hi @Boyan, Here are few links for you.

Use Databricks Repos and Repos API: Databricks Repos allow cloning whole git repositories in Databri.... This allows you to use a repository structure well known from IDEs in structuring your project, rely.... CI/CD pipelines on Azure DevOps can trigger Databricks Repos API to update this test project to the ....

Organize functions and unit tests: You can store functions and their unit tests outside of notebooks. This approach is beneficial as you can call these functions with and outside of notebooks. However, this approach requires Databricks Repos and increases the number of files to track and main....

Set up a CI/CD system: You can set up a continuous integration and continuous delivery or deployment....

[Boyan]

Hello @Kaniz ,

thank you for the links provided.

To run our production workloads we use python jobs and not notebooks. Most of the time, we develop a new job locally against mocked/test datasets and than we run into the issue I described. We run integration tests against an Azure Databricks cluster much less often.

So my question really is, is there a way to install the Databricks fork of hyperopt - version 0.2.7+db1 locally?