cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
alonisser
by • Contributor
  • 8033 Views
  • 14 replies
  • 7 kudos

Failing to install a library from dbfs mounted storage (adls2) with pass through credentials cluster

We've setup a premium workspace with passthrough credentials cluster , while they do work and access my adls gen 2 storageI can't make it install a library on the cluster from there. and keeping getting"Library installation attempted on the driver no...

Data Engineering
Reply
  • 8033 Views
  • 14 replies
  • 7 kudos
View Replies
Latest Reply
alonisser
Contributor
  • 7 kudos

Sorry I can't figure this out, the link you've added is irrelevant for passthrough credentials, if we add it the cluster won't be passthrough, Is there a way to add this just for a specific folder? while keeping passthrough for the rest?

  • 7 kudos
13 More Replies
Jeff1
by • Contributor II
  • 8679 Views
  • 4 replies
  • 4 kudos

Resolved! How to convert lat/long to geohash in databricks using geohashTools R library

I continues to receive a parsing error when attempting to convert lat/long data to a geohash in data bricks . I've tried two coding methods in R and get the same error.library(geohashTools)Method #1my_tbl$geo_hash <- gh_encode(my_tbl$Latitude, my_tbl...

Data Engineering
Reply
  • 8679 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Jeff1
Contributor II
  • 4 kudos

The problem was I was trying to run the gh_encode function on a Spark dataframe. I needed to collect the date into a R dataframe then run the function.

  • 4 kudos
3 More Replies
User16137833804
by • New Contributor III
  • 3170 Views
  • 3 replies
  • 1 kudos

How do I specify a version of a library different from the Databricks runtime?

Data Engineering
Reply
  • 3170 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Sebastian
Contributor
  • 1 kudos

the best solution is to store the .whl locally and do a pip install of the local whl while server boots up. this will freeze the library version. if you install from the pip it might impact your production work.

  • 1 kudos
2 More Replies
Anonymous
by • Not applicable
  • 1716 Views
  • 2 replies
  • 0 kudos

Issue loading spark Scala library

We have a proprietary spark scala library, which is necessary for me to do my work. We build a release version once a week and store it in a specific s3 location (so the most up-to-date prod version is always stored in the same place). But so far I c...

Data Engineering
Reply
  • 1716 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
User16857281974
Contributor
  • 0 kudos

Databrick's curriculum team solved this problem by creating our own Maven repo and it's easier than it sounds. To do this, we took an S3 bucket, converted it to a public website, allowing for standard file downloads, and then within that bucket creat...

  • 0 kudos
1 More Replies
alexott
by • Valued Contributor II
  • 2020 Views
  • 1 replies
  • 0 kudos

What libraries could be used for unit testing of the Spark code?

We need to add unit test cases for our code that we're writing using the Scala in Python. But we can't use the calls like `assertEqual` for comparing the content of DataFrames. Are any special libraries for that?

Data Engineering
Reply
  • 2020 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
alexott
Valued Contributor II
  • 0 kudos

There are several libraries for Scala and Python that help with writing unit tests for Spark code.For Scala you can use following:Built-in Spark test suite - it's designed to test all parts of Spark. It supports RDD, Dataframe/Dataset, Streaming API...

  • 0 kudos
jose_gonzalez
by • Moderator
  • 1083 Views
  • 2 replies
  • 0 kudos

Cluster goes unresponsive after installing a library

Right after I install a library in my cluster, my cluster goes unresponsive and nothing runs. How to solve this issue?

Data Engineering
Reply
  • 1083 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

it is a standard cluster. It is happening for all libraries. is there a way to debug or show the errors messages if any?

  • 0 kudos
1 More Replies
Labels