cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User15787040559
by • New Contributor III
  • 1513 Views
  • 2 replies
  • 1 kudos

How can I get Databricks notebooks to stop cutting off the explain plans?

(since Spark 3.0)Dataset.queryExecution.debug.toFilewill dump the full plan to a file, without concatenating the output as a fully materialized Java string in memory.

Data Engineering
Reply
  • 1513 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
dazfuller
Contributor III
  • 1 kudos

Notebooks really aren't the best method of viewing large files. Two methods you could employ areSave the file to dbfs and then use databricks CLI to download the fileUse the web terminalIn the web terminal option you can do something like "cat my_lar...

  • 1 kudos
1 More Replies
aladda
by • Honored Contributor II
  • 627 Views
  • 1 replies
  • 0 kudos

Can Databricks notebooks be hosted in S3 or object storage?

Data Engineering
Reply
  • 627 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

Databricks notebooks can be exported and stored in S3 or any other object storage. The internal storage of the databricks notebook cannot be changed or configured. The implementation is internal to Databicks control plane and not user configurable.

  • 0 kudos
Anonymous
by • Not applicable
  • 863 Views
  • 1 replies
  • 0 kudos

Saving charts in S3

Can you save the charts that you create in Databricks notebooks as an image file to s3? For example I'm viewing the dataframe using display(df) and then using the plot button to actually create the graph

Data Engineering
Reply
  • 863 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

Would rendering visualizations via plotly and saving them to s3 work ?

  • 0 kudos
aladda
by • Honored Contributor II
  • 681 Views
  • 1 replies
  • 0 kudos

Resolved! Can I use git or other git based version control systems to manage Databricks notebooks?

Data Engineering
Reply
  • 681 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Honored Contributor II
  • 0 kudos

There’s two places to leverage Github for content management and version control in DatabricksRepos for Git integration - Repos are folders whose contents are co-versioned together by syncing them to a remote Git repository. Repos can contain only Da...

  • 0 kudos
Labels