cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
nav
by • New Contributor II
  • 4003 Views
  • 8 replies
  • 0 kudos

R packages not getting installed on cluster when creating cluster from dockerfile

I'm trying to use dockerfile to create a cluster which has Robyn (https://facebookexperimental.github.io/Robyn/) and other R libraries installed. But it is failing to install the R libraries to the cluster. When I run the container in interactive mod...

Data Engineering
Reply
  • 4003 Views
  • 8 replies
  • 0 kudos
View Replies
Latest Reply
workingtogetdbw
New Contributor II
  • 0 kudos

What there has been no answer here!  @Debayan Mukherjee​ @Vartika Nain​ So I am running into this same problem as the idea of having to wait 45 minutes for libraries to install is absolutely wild as well as I have done everything outside of working w...

  • 0 kudos
7 More Replies
chandan_a_v
by • Valued Contributor
  • 8641 Views
  • 8 replies
  • 6 kudos

Resolved! logging.basicConfig not creating a file in Databricks

Hi,I am using the logger to log some parameters in my code and I want to save the file under DBFS. But for some reason the file is not getting created under DBFS. If I clear the state of the notebook and check the DBFS dir then file is present. Pleas...

Data Engineering
Reply
  • 8641 Views
  • 8 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Perhaps PyCharm sets a different working directory, meaning the file ends up in another place. Try providing a full path.

  • 6 kudos
7 More Replies
SailajaB
by • Valued Contributor III
  • 7027 Views
  • 12 replies
  • 4 kudos

Resolved! JSON validation is getting failed after writing Pyspark dataframe to json format

Hi We have to convert transformed dataframe to json format. So we used write and json format on top of final dataframe to convert it to json. But when we validating the output json its not in proper json format.Could you please provide your suggestio...

Data Engineering
Reply
  • 7027 Views
  • 12 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@Sailaja B​ - Does @Aman Sehgal​'s most recent answer help solve the problem? If it does, would you be happy to mark their answer as best?

  • 4 kudos
11 More Replies
Labels