cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
priyak
by • New Contributor III
  • 9041 Views
  • 8 replies
  • 3 kudos

Resolved! Multiple versions of custom libraries on the cluster

Using the install_libraries API, I installed a custom Python whl file on a running cluster. For certain types of requests, we have a requirement to install a different version of the same custom whl file in the running cluster. My problem is that uni...

Data Engineering
Reply
  • 9041 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
bkapers
New Contributor II
  • 3 kudos

Why does databricks forum not filter out these spambot AI replies ?  If one wants an AI chatbot to guess an answer their question, they are free to directly ask ChatGPT, etc.  A community forum is for human-written answers based on actual first-hand ...

  • 3 kudos
7 More Replies
AyushModi038
by • New Contributor III
  • 14597 Views
  • 8 replies
  • 10 kudos

Library installation in cluster taking a long time

I am trying to install "pycaret" libraray in cluster using whl file.But it is creating conflict in the dependency sometimes (not always, sometimes it works too.) ​My questions are -1 - How to install libraries in cluster only single time (Maybe from ...

Data Engineering
Reply
  • 14597 Views
  • 8 replies
  • 10 kudos
View Replies
Latest Reply
Spencer_Kent
New Contributor III
  • 10 kudos

@Retired_modWhat about question #1, which is what subsequent comments to this thread have been referring to? To recap the question: is it possible for "cluster-installed" libraries to be cached in such a way that they aren't completely reinstalled ev...

  • 10 kudos
7 More Replies
Kaijser
by • New Contributor II
  • 2927 Views
  • 1 replies
  • 2 kudos

Installing private python Azure DevOps repository without revealing personal access token in pyproject.toml

I want to install a .whl file on my Databricks cluster which includes a private Azure DevOps repository as a dependency in its pyproject.toml file, i.e.:[project] name = "test" description = "test_description." version = "0.1.0" authors = [ { name ...

Data Engineering
Reply
  • 2927 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Aaron Kaijser​  Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
Labels