cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
lei_armstrong
by • New Contributor II
  • 9319 Views
  • 6 replies
  • 6 kudos

Resolved! Executing Notebooks - Run All Cells vs Run All Below

Due to dependencies, if one of our cells errors then we want the notebook to stop executing.We've noticed some odd behaviour when executing notebooks depending on if "Run all cells in this notebook" is selected from the header versus "Run All Below"....

Data Engineering
Reply
  • 9319 Views
  • 6 replies
  • 6 kudos
View Replies
Latest Reply
sukanya09
New Contributor II
  • 6 kudos

Has this been implemented? I have created a job using notebook. My notebook has 6 cells and if the code in first cell fails it should not run the rest of the cells 

  • 6 kudos
5 More Replies
Oliver_Angelil
by • Valued Contributor II
  • 7875 Views
  • 2 replies
  • 2 kudos

Resolved! Cell by cell execution of notebooks with VS code

I have the Databricks VS code extension setup to develop and run jobs remotely. (with Databricks Connect).I enjoy working on notebooks within the native Databricks workspace, especially for exploratory work because I can execute blocks of code step b...

Data Engineering
Reply
  • 7875 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
awadhesh14
New Contributor II
  • 2 kudos

Hi Folks,Is there a version upgrade for the resolution to this?

  • 2 kudos
1 More Replies
Ank
by • New Contributor II
  • 8288 Views
  • 5 replies
  • 6 kudos

Why am I getting NameError name ' ' is not defined in another cell?

I defined a dictionary variable Dict, populated it, and print(dict) in the first cell of my notebook. In the next cell, I executed the command print(dict) again. However, this time it gave me an error NameError: name 'Dict is not definedHow can that ...

Data Engineering
Reply
  • 8288 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
erigaud
Honored Contributor
  • 6 kudos

Running pip install restarts the interpreter, meaning that any variable defined prior to the pip install is lost, so indeed the solution is so run the pip install first, or better is to add the library you want to installl directly to the cluster con...

  • 6 kudos
4 More Replies
Volkan_Gumuskay
by • New Contributor III
  • 6648 Views
  • 6 replies
  • 3 kudos

Resolved! Is there a way to run a single or selected lines in a notebook?

Assume we have a given cellprint('A') print('B') print('C')I want to run only the below line.print('B')Obviously, I can seperate the cell into three and run the one I want, but this is timely. This is a feature I use so often (e.g. in pycharm) and wo...

Data Engineering
Reply
  • 6648 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
Tharun-Kumar
Databricks Employee
  • 3 kudos

@Volkan_Gumuskay This is also available as an option in the notebook run options.

  • 3 kudos
5 More Replies
Divya_Bhadauria
by • New Contributor II
  • 667 Views
  • 0 replies
  • 0 kudos

Number of rows displayed in sql cell

Even though the default limit on rows displayed is 10,000, the SQL cell is showing rows less than the limit when my resultant has more rows than 10k.It should alteast show the default limit .

Data Engineering
Reply
  • 667 Views
  • 0 replies
  • 0 kudos
sanq
by • New Contributor II
  • 3956 Views
  • 2 replies
  • 7 kudos

what formatter is used to format SQL cell in databricks

Databricks launched formatter Black which formats python cells, I can also see SQL cell getting formatted, but not sure which formatter is being used for SQL cell formatting. No clarity given on docs.

Data Engineering
Reply
  • 3956 Views
  • 2 replies
  • 7 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 7 kudos

Hi @SANJAY qq​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 7 kudos
1 More Replies
KVNARK
by • Honored Contributor II
  • 3223 Views
  • 2 replies
  • 4 kudos

Resolved! enabling the cell in Databricks.

Suppose if the query is long one and its commented due to some issues later I wanted to run that cell, is there any shortcut to enable the entire cell. The cell is with 800 lines of code and each line is commented with # symbol and I want to enable i...

Data Engineering
Reply
  • 3223 Views
  • 2 replies
  • 4 kudos
View Replies
Latest Reply
pvignesh92
Honored Contributor
  • 4 kudos

@KVNARK .​ Hi,If I understand correctly, you want to either comment or disable comment on your entire cell using a Shortcut.If that's the case, To do for the whole cell -> Do a Ctrl + A, then you can use Ctrl + / in Windows. It will add # to all your...

  • 4 kudos
1 More Replies
moski
by • New Contributor II
  • 7491 Views
  • 8 replies
  • 7 kudos

Databricks short cut to split a cell

Is there a shortcut to split a cell into two in Dtabricks notebook as in Jupiter notebook? in jupyter notebook it is Shift/Ctr/-

Data Engineering
Reply
  • 7491 Views
  • 8 replies
  • 7 kudos
View Replies
Latest Reply
Harshjot
Contributor III
  • 7 kudos

 Hi @mundy Jim​ / All, Attached are two snapshots so first snapshot with one cell if pressed Ctrl+Alt+Minus split into two.  

  • 7 kudos
7 More Replies
ossinova
by • Contributor II
  • 1643 Views
  • 1 replies
  • 3 kudos

Resolved! Shortcut for changing cell language (adding magic command)

Is there an option or shortcut (or is it in the pipeline) to quickly change / insert a cell to a specific language in Databricks?Triggering B + P would for instance add a new cell below with %pythonTriggering M would for instance change that cell to ...

Data Engineering
Reply
  • 1643 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 3 kudos

I've passed this message along to the UX UI Feedback team.

  • 3 kudos
ThomasKastl
by • Contributor
  • 4849 Views
  • 6 replies
  • 5 kudos

Resolved! Databricks runs cell, but stops output and hangs afterwards.

tl;dr: A cell that executes purely on the head node stops printed output during execution, but output still shows up in the cluster logs. After execution of the cell, Databricks does not notice the cell is finished and gets stuck. When trying to canc...

Data Engineering
Reply
  • 4849 Views
  • 6 replies
  • 5 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 5 kudos

As that library work on pandas problem can be that it doesn't support pandas on spark. On the local version, you probably use non-distributed pandas. You can check behavior by switching between:import pandas as pd import pyspark.pandas as pd

  • 5 kudos
5 More Replies
THIAM_HUATTAN
by • Valued Contributor
  • 1661 Views
  • 2 replies
  • 0 kudos

Resolved! Save data from Spark DataFrames to TFRecords

https://docs.microsoft.com/en-us/azure/databricks/_static/notebooks/deep-learning/tfrecords-save-load.htmlI could not run the Cell # 2java.lang.ClassNotFoundException: --------------------------------------------------------------------------- Py4JJ...

Data Engineering
Reply
  • 1661 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Hi @THIAM HUAT TAN​,Which DBR version are you using? are you using the ML runtime?

  • 0 kudos
1 More Replies
Labels