cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
User15787040559
by • Databricks Employee
  • 2153 Views
  • 1 replies
  • 1 kudos

How can I get Databricks notebooks to stop cutting off the explain plans?

(since Spark 3.0)Dataset.queryExecution.debug.toFilewill dump the full plan to a file, without concatenating the output as a fully materialized Java string in memory.

Data Engineering
Reply
  • 2153 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
dazfuller
Contributor III
  • 1 kudos

Notebooks really aren't the best method of viewing large files. Two methods you could employ areSave the file to dbfs and then use databricks CLI to download the fileUse the web terminalIn the web terminal option you can do something like "cat my_lar...

  • 1 kudos
aladda
by • Databricks Employee
  • 981 Views
  • 1 replies
  • 0 kudos

Can Databricks notebooks be hosted in S3 or object storage?

Data Engineering
Reply
  • 981 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

Databricks notebooks can be exported and stored in S3 or any other object storage. The internal storage of the databricks notebook cannot be changed or configured. The implementation is internal to Databicks control plane and not user configurable.

  • 0 kudos
Anonymous
by • Not applicable
  • 1437 Views
  • 1 replies
  • 0 kudos

Saving charts in S3

Can you save the charts that you create in Databricks notebooks as an image file to s3? For example I'm viewing the dataframe using display(df) and then using the plot button to actually create the graph

Data Engineering
Reply
  • 1437 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

Would rendering visualizations via plotly and saving them to s3 work ?

  • 0 kudos
aladda
by • Databricks Employee
  • 1239 Views
  • 1 replies
  • 0 kudos

Resolved! Can I use git or other git based version control systems to manage Databricks notebooks?

Data Engineering
Reply
  • 1239 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 0 kudos

There’s two places to leverage Github for content management and version control in DatabricksRepos for Git integration - Repos are folders whose contents are co-versioned together by syncing them to a remote Git repository. Repos can contain only Da...

  • 0 kudos
Labels