We have written few python functions(methods within a class) and packaged them as a wheel library.

In the as-is situation we use to install that wheel library in All-Purpose cluster that we already have created.

It works fine.

In the to-be situtation(Delta Live Tables) we want this wheel library to be installed part of the Delta live pipeline execution, because when DLT pipeline runs it creates its own Job Cluster.

We use lot of python functions to do the transformations between Silver and Gold layer.

Hence we want the wheel library (which has all the UDF’s) to be installed in the Job Cluster which DLT pipeline creates.

When we execute %pip install <wheel library location in DBFS> as a first step in the DLT notebook, it does not seem to work.

But when we have %pip install numpy it works.

Its important for us to have the wheel library installed in the job cluster created by DLT pipeline.

Are we missing something?

Thanks