- 833 Views
- 1 replies
- 2 kudos
Best Suited Language To Parallelize Notebook
I would like to know if we are writing code for parallel execution on notebook which language is best suited for that Python or Scala.
- 833 Views
- 1 replies
- 2 kudos
- 2 kudos
You need to test in Python and scala based on the complexity one of it outperforms the other. In few cases Python was faster where as in other Scala. It is all about the efficiency of the code
- 2 kudos
- 5751 Views
- 2 replies
- 0 kudos
`moduleNotFoundError` when attempting to enable a jupyter notebook extension
I am running a set of commands and to run `pydeck` on jupyter notebook as per the documentation here: https://pydeck.gl/installation.html#enabling-pydeck-for-jupyterHowever, it throws an `moduleNotFoundError` exception. !pip install pydeck !jupyter n...
- 5751 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Keval Shah​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 0 kudos
- 2349 Views
- 3 replies
- 5 kudos
How to exit the entire job in the orchestration scenario?
Hi, can anybody answer this question I posted on StackOverflow? https://stackoverflow.com/questions/73314048/databricks-how-to-exit-the-entire-job-in-the-notebooks-orchestration-scenario
- 2349 Views
- 3 replies
- 5 kudos
- 5 kudos
@Vidula Khanna​ @Vidula Khanna​ We are experiencing the same issue in our Workflows and I was wondering if there has been any update.We need the functionality to call a method similar to `dbutils.notebook.exit` in a notebook that will cancel the exec...
- 5 kudos
- 3632 Views
- 3 replies
- 5 kudos
Support of running multiple cells at a time in databricks notebook Hi all,Now databricks notebook supports parallel run of commands in a single notebo...
Support of running multiple cells at a time in databricks notebookHi all,Now databricks notebook supports parallel run of commands in a single notebook that will help run ad hoc queries simultaneously without creating a separate notebook.Once you run...
- 3632 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Ajay Pandey​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so w...
- 5 kudos
- 34215 Views
- 9 replies
- 6 kudos
Resolved! How to access the result of a %sql cell from python
I see the way to move from python to sql is to create a temp view, and then access that dataframe from sql, and in a sql cell. Now the question is, how can I have a %sql cell with a select statement in it, and assign the result of that statement to ...
- 34215 Views
- 9 replies
- 6 kudos
- 6 kudos
Results from an SQL cell are available as a Python DataFrame. The Python DataFrame name is _sqldf.To save the DataFrame, run this code in a Python cell:df = _sqldfKeep in mind that the value in _sqldf is held in memory and will be replaced with the m...
- 6 kudos
- 10338 Views
- 2 replies
- 0 kudos
Combine Python + R in data manipulation in Databricks Notebook
Want to combine Py + Rfrom pyspark.sql import SparkSessionspark = SparkSession.builder.appName("CreateDataFrame").getOrCreate()# Create a sample DataFramedata = [("Alice", 25), ("Bob", 30), ("Charlie", 35), ("Oscar",36), ("Hiromi",41), ("Alejandro", ...
- 10338 Views
- 2 replies
- 0 kudos
- 0 kudos
@Oscar CENTENO MORA​ :To combine Py and R in a Databricks notebook, you can use the magics command %python and %rto switch between Python and R cells. Here's an example of how to create a Spark DataFrame in Python and then use it in R:from pyspark.sq...
- 0 kudos
- 1887 Views
- 2 replies
- 0 kudos
Run pytest inside repos and store the results in dbfs
Hi everyone!I am trying to run pytest inside a notebook on repos and store the results inside dbfs but i am getting an error stating permission denied, does anyone know why this happens and the solution. Error:
- 1887 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @João Peixoto​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...
- 0 kudos
- 2710 Views
- 3 replies
- 1 kudos
How to get url of the dashboard corresponding to a notebook?
I have several notebooks and their corresponding dashboards. I was able to get the url of the notebooks via the workspace api by concatenating the static part of the url with the object ids, but I am struggling to find a way to do the same for their ...
- 2710 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Akshaya Choudhary​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...
- 1 kudos
- 3697 Views
- 5 replies
- 3 kudos
Give customized notebook run id/name in console using dbutils.notebook.run()
I am running below code in Azure databricks:Now the running status shows as highlighted in yellow. I want to give a custom name to this running status instead of Notebook job #<job id>. This will help me identify which status is for which notebook so...
- 3697 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Gaurav Tanwar​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I suggest...
- 3 kudos
- 3195 Views
- 1 replies
- 3 kudos
TimestampFormat issue
The databricks notebook failed yesterday due to timestamp format issue. error:"SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '2022-08-10 00:00:14.2760000' in the new parser. You can set spark.s...
- 3195 Views
- 1 replies
- 3 kudos
- 3 kudos
You must have solved this issue by now but for the sake of those that encounter this again, here's the solution that worked for me:spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
- 3 kudos
- 2673 Views
- 4 replies
- 16 kudos
ipywidget not able to start running another notebook
I am trying to add a button in a notebook to trigger an execution of another notebook, but it does not respond to the command. Any idea why? The run command works if I run in a separate cell
- 2673 Views
- 4 replies
- 16 kudos
- 16 kudos
Did you end up finding an answer to this? I am running into the same problem
- 16 kudos
- 9691 Views
- 13 replies
- 14 kudos
Resolved! How to find the last modified date of a notebook?
I would like to find the notebooks that are not required and not being used and then I can review and delete them. If there is a way to find last modified date of a notebook programmatically then I can get a list of notebooks, which I can review and ...
- 9691 Views
- 13 replies
- 14 kudos
- 14 kudos
Hi @Naveen Kumar Madas​ you can go through below code block%shls -lt /dbfs/
- 14 kudos
- 5893 Views
- 3 replies
- 5 kudos
Notebook dropdown widget
I have created a dropdown (say B) in my notebook whose input depend on dropdown( say B). So if select some value in dropdown A, it corresponding value appears in B dropdown & i'm selecting one amongst it. Now if i change the value in dropdown A, then...
- 5893 Views
- 3 replies
- 5 kudos
- 5 kudos
If the previously selected value of B is not meant to be in the list of values for newly selected dropdown A value, then you could set a default value (ie: 'No selection') that the B dropdown should have when first created. In a method to define how ...
- 5 kudos
- 1507 Views
- 2 replies
- 0 kudos
- 1507 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Youssef Mrini​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...
- 0 kudos
- 1445 Views
- 1 replies
- 0 kudos
- 1445 Views
- 1 replies
- 0 kudos
- 0 kudos
The UNPIVOT clause is now supported by Databricks SQL. Use the UNPIVOT clause to rotate columns of a table-valued expression into column values https://docs.databricks.com/sql/language-manual/sql-ref-syntax-qry-select-unpivot.html
- 0 kudos