DLT UC for ML use cases

[karthik_p]

Hi Team,

we have use case to run ML use case using DLT UC, we are facing library issues 

Error: INVALID ARGUMENT: no module named ‘importlib_metadata’

we installed them manually by passing import in notebooks, but we have lot of missing libraries and we are seeing few of them getting failed 

how to overcome this in DLT UC, as we have limitations in terms of single user access clusters. How can we handle ML use cases

[Kaniz]

Hi @karthik_p, The error no module named 'importlib_metadata' is usually due to the absence of the importlib_metadata library in your Python environment. You can install it manually using the command pip install importlib-metadata. 

However, note that since Python 3.8, the importlib-metadata third-party library has been merged into.... 

So, you can use it like this:

 from importlib.metadata import distribution,metadata,version.

For handling missing libraries in DLT UC, Delta Live Tables supports external dependencies in your pipelines. 

You can use the %pip install command to install packages for all source files in a pipeline. Every Python notebook included in the pipeline shares a library environment and has access to all in....

For handling ML use cases with single user access clusters, Databricks recommends using service prin.... Clusters configured with Single User access mode have full access to DBFS, including all files in th....

[karthik_p]

@Kaniz do we have any support of shared cluster for ML in near roadmap please