cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
houstonamoeba
by • New Contributor III
  • 3972 Views
  • 7 replies
  • 1 kudos

Resolved! examples on python sdk for install libraries

Hi Everyone,I'm planning to use databricks python cli "install_libraries"can some one pls post examples on function install_libraries https://github.com/databricks/databricks-cli/blob/main/databricks_cli/libraries/api.py

Data Engineering
Reply
  • 3972 Views
  • 7 replies
  • 1 kudos
View Replies
Latest Reply
Loop-Insist
New Contributor II
  • 1 kudos

Here you go using Python SDKfrom databricks.sdk import WorkspaceClientfrom databricks.sdk.service import computew = WorkspaceClient(host="yourhost", token="yourtoken")# Create an array of Library objects to be installedlibraries_to_install = [compute...

  • 1 kudos
6 More Replies
carlosst01
by • New Contributor II
  • 1870 Views
  • 2 replies
  • 2 kudos

Resolved! Running Libraries and/or modules in Databricks' lifecycle?

Hi, i have had this question for some weeks and didn't find any information about the topic. Specifically, my doubt is: what is the 'lifecycle' or cycle or steps to be able to use a new Python library in Databricks in terms of compatibility? For exam...

Data Engineering
Reply
  • 1870 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Carlos Caravantes​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 2 kudos
1 More Replies
ArturoNuor
by • New Contributor III
  • 4320 Views
  • 3 replies
  • 0 kudos

Resolved! Unable to install R geospatial libraries raster, terra, sf, ncdf4, etc

When trying to install any of this R libraries from a cmd cell/block on a notebook, or from the UI in the cluster I receive the same error,seeming that are unable to install dependecies.Warning in utils::install.packages(pkgs, ...) : installation of ...

Data Engineering
Reply
  • 4320 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
ArturoNuor
New Contributor III
  • 0 kudos

For the next soul looking for an answer, I managed to solve the issue with the next 2 Init scripts, it gets tricky in the apt or apt-get, that was the issue, sometimes it did update, sometimes it didn't, making it possible to find libmysqlclient21.1)...

  • 0 kudos
2 More Replies
jwilliam
by • Contributor
  • 1710 Views
  • 2 replies
  • 2 kudos

Resolved! Does libraries installation happen on Data Plane or Control Plane?

Currently, when I install libraries on my clusters. This errors happens:WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol...

Data Engineering
Reply
  • 1710 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Sivaprasad1
Valued Contributor II
  • 2 kudos

@John William​ : Yeah that's true. All the clusters will be residing in the data plane.

  • 2 kudos
1 More Replies
isaac_gritz
by • Databricks Employee
  • 3320 Views
  • 6 replies
  • 8 kudos

Library Dependency

How to Install Libraries on DatabricksYou can install libraries in Databricks at the cluster level for libraries commonly used on a cluster, at the notebook-level using %pip, or using global init scripts when you have libraries that should be install...

Data Engineering
Reply
  • 3320 Views
  • 6 replies
  • 8 kudos
View Replies
Latest Reply
Chris_Shehu
Valued Contributor III
  • 8 kudos

It can be a risky to install libraries without any sort of oversite/security structure to ensure those libraries have no vulnerabilities. I think more caution needs to be added to the wording of these documents to express that. All of the libraries w...

  • 8 kudos
5 More Replies
yannickmo
by • New Contributor III
  • 6343 Views
  • 7 replies
  • 14 kudos

Resolved! Adding JAR from Azure DevOps Artifacts feed to Databricks job

Hello,We have some Scala code which is compiled and published to an Azure DevOps Artifacts feed.The issue is we're trying to now add this JAR to a Databricks job (through Terraform) to automate the creation.To do this I'm trying to authenticate using...

Data Engineering
Reply
  • 6343 Views
  • 7 replies
  • 14 kudos
View Replies
Latest Reply
alexott
Databricks Employee
  • 14 kudos

As of right now, Databricks can't use non-public Maven repositories as resolving of the maven coordinates happens in the control plane. That's different from the R & Python libraries. As workaround you may try to install libraries via init script or ...

  • 14 kudos
6 More Replies
nikgoel95
by • New Contributor II
  • 1295 Views
  • 3 replies
  • 1 kudos

What's the be​at way to define the libraries for cluster as it always take a lot of time for me.

What's the be​at way to define the libraries for cluster as it always take a lot of time for me.

Data Engineering
Reply
  • 1295 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Sivaprasad1
Valued Contributor II
  • 1 kudos

@Nikunj Goel​ : Please refer to the below doc the workspace library might help on thishttps://docs.databricks.com/libraries/workspace-libraries.html#workspace-libraries

  • 1 kudos
2 More Replies
pbezz
by • New Contributor III
  • 6869 Views
  • 5 replies
  • 9 kudos

Visualisation libraries does not work on Databricks

Why is it that certain Python visualisation libraries do not work on Databricks? I am trying to install (via pip) and work with some data visualisation libraries - they work perfectly in a normal Jupyter Notebook but not on a Databricks notebook envi...

Data Engineering
Reply
  • 6869 Views
  • 5 replies
  • 9 kudos
View Replies
Latest Reply
pbezz
New Contributor III
  • 9 kudos

No switched to using html widgets.

  • 9 kudos
4 More Replies
William_Scardua
by • Valued Contributor
  • 2171 Views
  • 1 replies
  • 2 kudos

Resolved! Best way to encrypt PII data

Hi guys, I have around 600GB per load, in you opnion, what is the best way to encrypt PII data in terms of performance ? (lib, cluster type, etc.)Thank youWilliam

Data Engineering
Reply
  • 2171 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Prabakar
Databricks Employee
  • 2 kudos

Hello @William Scardua​ please check if the blog helps you.https://databricks.com/blog/2020/11/20/enforcing-column-level-encryption-and-avoiding-data-duplication-with-pii.html

  • 2 kudos
reedzhang
by • New Contributor III
  • 3884 Views
  • 4 replies
  • 3 kudos

Resolved! uninstalled libraries continue to get installed on cluster startup

We have been trying to update some library versions by uninstalling the old versions and installing new ones. However, the old libraries continue to get installed on cluster startup despite not showing up in the "libraries" tab of the cluster page. W...

Data Engineering
Reply
  • 3884 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
reedzhang
New Contributor III
  • 3 kudos

The issue seemed to go away on its own. At some point the libraries page started showing what was getting installed to the cluster, and removing libraries from the page caused them to stop getting installed on cluster startup. I'm guessing there was ...

  • 3 kudos
3 More Replies
RasmusOlesen
by • New Contributor III
  • 4592 Views
  • 5 replies
  • 1 kudos

Resolved! ciso8601 library stopped installing out of the blue on DB clusters

We have multiple DB clusters (6.4 Extended Support) that have not changed in terms of libs installed or nodes etc. Sudden from one day to the other, after a cluster restart August 7th, they stopped installing ciso8601 lib as they would usually. Anyb...

Data Engineering
Reply
  • 4592 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
RasmusOlesen
New Contributor III
  • 1 kudos

Just to close this old qustion:We solved this by switching to a PEP517 free pip install, using the a Global Init Script:/databricks/python/bin/pip install ciso8601 --disable-pip-version-check --no-use-pep517Now it works for us.

  • 1 kudos
4 More Replies
kjoth
by • Contributor II
  • 13434 Views
  • 5 replies
  • 5 kudos

Resolved! Databricks default python libraries list & version

We are using data-bricks. How do we know the default libraries installed in the databricks & what versions are being installed. I have ran pip list, but couldn't find the pyspark in the returned list.

Data Engineering
Reply
  • 13434 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 5 kudos

Hi @karthick J​ ,If you would like to see all the libraries installed in your cluster and the version, then I will recommend to check the "Environment" tab. In there you will be able to find all the libraries installed in your cluster.Please follow t...

  • 5 kudos
4 More Replies
IkramMecheri
by • New Contributor II
  • 11609 Views
  • 3 replies
  • 1 kudos

ImportError: No module named 'bs4'

Hi, I would like to do some web scrapping, however I am unable to import the libraries I traditionally use for that task import requests from bs4 import BeautifulSoup

Data Engineering
Reply
  • 11609 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Hayley
Databricks Employee
  • 1 kudos

Did you try `%pip install bs4 `requests is standard in the databricks runtime, so you don't have to install it.

  • 1 kudos
2 More Replies
morganmazouchi
by • Databricks Employee
  • 7238 Views
  • 6 replies
  • 4 kudos

Resolved! Is there a way that admins can restrict users to install libraries on clusters and notebooks?

Data Engineering
Reply
  • 7238 Views
  • 6 replies
  • 4 kudos
View Replies
Latest Reply
Sebastian
Contributor
  • 4 kudos

one way to manage is make the cluster permission only to can restart and then use an init script to install libraries on start up so that users wont install libraries on the fly.

  • 4 kudos
5 More Replies
User16830818524
by • New Contributor II
  • 1758 Views
  • 3 replies
  • 0 kudos

Resolved! Libraries in Databricks Runtimes

Is it possible to easily determine what libraries and which version are included in a specific DBR Version?

Data Engineering
Reply
  • 1758 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hello. My name is Piper and I'm one of the community moderators. One of the team members sent this information to me.This should be the correct path to check libraries installed with DBRs.https://docs.databricks.com/release-notes/runtime/8.3ml.html?_...

  • 0 kudos
2 More Replies
Labels