topic Re: Library Dependency in Data Engineering

Library Dependency

isaac_gritz — Tue, 23 Aug 2022 08:20:35 GMT

How to Install Libraries on Databricks

You can install libraries in Databricks at the cluster level for libraries commonly used on a cluster, at the notebook-level using %pip, or using global init scripts when you have libraries that should be installed on all clusters.

The Databricks ML runtime comes with many commonly used ML libraries pre-installed including sklearn, Tensorflow, and XGBoost (AWS | Azure | GCP)

You can learn more at this page (AWS | Azure | GCP). Let us know in the comments if you have any questions on library dependency!

Re: Library Dependency

Abishek — Thu, 25 Aug 2022 08:39:09 GMT

@Isaac Gritz Adding a few more docs links for > https://docs.databricks.com/libraries/index.html

Re: Library Dependency

isaac_gritz — Thu, 25 Aug 2022 08:42:02 GMT

Thanks Abishek!

Re: Library Dependency

Prabakar — Sat, 27 Aug 2022 03:35:21 GMT

Good post @Isaac Gritz and thanks @Abishek Subramanian for adding those links.

Re: Library Dependency

LearningDatabri — Sat, 27 Aug 2022 03:56:57 GMT

I saw couple of you posts and it seems that you are sharing the information from the public docs. instead of sharing the docs information you can post some errors and the troubleshooting and resolution steps. That will be much helpful. Thanks.

Re: Library Dependency

isaac_gritz — Sat, 27 Aug 2022 04:03:32 GMT

Hi team, I am a Databricks employee and I am posting common questions I have received from Databricks customers in the past and answers I would typically provide. I reached out to @Lindsay Olson to validate this approach. Happy to remove any posts or find a new forum/format if you would like.

Re: Library Dependency

Chris_Shehu — Sun, 25 Sep 2022 20:13:21 GMT

It can be a risky to install libraries without any sort of oversite/security structure to ensure those libraries have no vulnerabilities. I think more caution needs to be added to the wording of these documents to express that. All of the libraries we use go through a vetting process before they can actually be installed.