cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Chaitanya_Raju
by Honored Contributor
  • 3983 Views
  • 7 replies
  • 0 kudos
  • 3983 Views
  • 7 replies
  • 0 kudos
Latest Reply
Vartika
Moderator
  • 0 kudos

Hi @Ratna Chaitanya Raju Bandaru​Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best? If not, please tell us so we can help you.Thanks!

  • 0 kudos
6 More Replies
brickster_2018
by Esteemed Contributor
  • 3813 Views
  • 2 replies
  • 3 kudos

Resolved! Can I install notebook scoped JAR/Maven libraries?

The notebook scoped libraries are very handy. Is it possible to leverage the same for maven jars or application jars as well?

  • 3813 Views
  • 2 replies
  • 3 kudos
Latest Reply
Pratik_Ghosh
New Contributor II
  • 3 kudos

Any further update on this topic?

  • 3 kudos
1 More Replies
Volkan_Gumuskay
by New Contributor III
  • 5772 Views
  • 6 replies
  • 3 kudos

Resolved! Is there a way to run a single or selected lines in a notebook?

Assume we have a given cellprint('A') print('B') print('C')I want to run only the below line.print('B')Obviously, I can seperate the cell into three and run the one I want, but this is timely. This is a feature I use so often (e.g. in pycharm) and wo...

  • 5772 Views
  • 6 replies
  • 3 kudos
Latest Reply
Tharun-Kumar
Honored Contributor II
  • 3 kudos

@Volkan_Gumuskay This is also available as an option in the notebook run options.

  • 3 kudos
5 More Replies
chandan_a_v
by Valued Contributor
  • 7047 Views
  • 3 replies
  • 6 kudos

How to restart the Spark session within the notebook without reattaching the notebook?

Hi All,I want to run an ETL pipeline in a sequential way in my DB notebook. If I run it without resetting the Spark session or restarting the cluster I am getting a data frame key error. I think this might be because of the Spark cache because If I r...

  • 7047 Views
  • 3 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Is there a solution to the above problem? I also would like to restart SparkSession to free my cluster's resources, but when callingspark.stop()the notebook automatically detach and the following error occurs:The spark context has stopped and the dri...

  • 6 kudos
2 More Replies
Silán
by New Contributor II
  • 1621 Views
  • 3 replies
  • 4 kudos

Resolved! Kept outputs

Hi everyone,I was wondering if perhaps someone of you could tell me which kinds of outputs are kept in a notebook after the cluster to which it is attached is terminated... Actually, I am asking it especially because I lost some visualization that I ...

  • 1621 Views
  • 3 replies
  • 4 kudos
Latest Reply
Silán
New Contributor II
  • 4 kudos

Great. Thanks a lot.

  • 4 kudos
2 More Replies
elikvar
by New Contributor III
  • 17467 Views
  • 9 replies
  • 9 kudos

Cluster occasionally fails to launch

I have a daily running notebook that occasionally fails with the error:"Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxx is in un...

  • 17467 Views
  • 9 replies
  • 9 kudos
Latest Reply
Lebreton
New Contributor II
  • 9 kudos

hello any update on this issue ?We have the same problem and no logs to investigate (even in dbfs when we activate the logging)Unexpected failure while waiting for the cluster (<id of our cluster>) to be ready: Cluster <id of our cluster> is in unexp...

  • 9 kudos
8 More Replies
cmilligan
by Contributor II
  • 760 Views
  • 1 replies
  • 1 kudos

Return notebook path from job that is run remotely from the repo

I'm wanting to set up some email alerts for issues in the data as a part of a job run. I am wanting to point the user to the notebook that the issue occurred in. I think this would be simple enough but another layer is that the job is going to be run...

  • 760 Views
  • 1 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, Could you please clarify what do you mean by return the file from the remote repo?Please tag @Debayan​ with your next response which will notify me, Thank you!

  • 1 kudos
Papa_PSL
by New Contributor
  • 1317 Views
  • 2 replies
  • 1 kudos

I want to import a notebook

Want to import a note book, facing this following error:Importation fails: could not deserialize: Unkown language htm

  • 1317 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @BITAIGOT Guy-Serge​ We haven't heard from you since the last response from @Debayan Mukherjee​ â€‹, and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be hel...

  • 1 kudos
1 More Replies
Mumrel
by Contributor
  • 2094 Views
  • 2 replies
  • 2 kudos

Resolved! Error 95 when importing one Notebook into another

When I follow the instructions Modularize your code using files I get the following error:I am on azure, use DBRT 12.2 LTS, use ADLS as storage, I am happy to provide more details if needed. My research suggest that the reason is that the dfbs fuse...

image
  • 2094 Views
  • 2 replies
  • 2 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

import works for .py files..%run is for notebooks.is lib a .py file or a notebook?

  • 2 kudos
1 More Replies
Muskan
by New Contributor III
  • 5279 Views
  • 9 replies
  • 1 kudos

Unable to launch notebook

I have created a 12.2 LTS cluster and trying to launch notebook attached to this cluster. But unable to launch, it is not giving any error instead it is still showing the same home page.

  • 5279 Views
  • 9 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Muskan Bansal​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...

  • 1 kudos
8 More Replies
shaunangcx
by New Contributor II
  • 2828 Views
  • 3 replies
  • 0 kudos

Resolved! Command output disappearing (Not sure what's the root cause)

I have a workflow which will run every month and it will create a new notebook containing the outputs from the main notebook. However, after some time, the outputs from the created notebook will disappear. Is there anyway I can retain the outputs?

  • 2828 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Shaun Ang​ :There are a few possible reasons why the outputs from the created notebook might be disappearing:Notebook permissions: It's possible that the user or service account running the workflow does not have permission to write to the destinati...

  • 0 kudos
2 More Replies
Anonymous
by Not applicable
  • 536 Views
  • 1 replies
  • 2 kudos

www.databricks.com

Dear Community - @Youssef Mrini​ will answer all your questions on April 19, 2023 from 9:00am to 10:00am GMT during the Databricks EMEA Office Hours.Make sure to join this amazing 'Ask Me Anything' session by Databricks - https://www.databricks.com/r...

  • 536 Views
  • 1 replies
  • 2 kudos
Latest Reply
youssefmrini
Honored Contributor III
  • 2 kudos

It was a successful office hours. Make sure to join the next one.

  • 2 kudos
Data_Engineer3
by Contributor III
  • 10076 Views
  • 4 replies
  • 5 kudos

How can i use the same spark session from onenotebook to another notebook in databricks

I want to use the same spark session which created in one notebook and need to be used in another notebook in across same environment, Example, if some of the (variable)object got initialized in the first notebook, i need to use the same object in t...

  • 10076 Views
  • 4 replies
  • 5 kudos
Latest Reply
Manoj12421
Valued Contributor II
  • 5 kudos

You can use %run and then use the location of the notebook - %run "/folder/notebookname"

  • 5 kudos
3 More Replies
Anonymous
by Not applicable
  • 6705 Views
  • 1 replies
  • 1 kudos

Testing framework using Databricks Notebook and Pytest.

Hi Friends,I am designing a Testing framework using Databricks and pytest. Currently stuck with report generation, that is generating blank with only default parameters only .for ex :-testsuites><testsuite name="pytest" errors="0" failures="0" skippe...

  • 6705 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Vijaya Palreddy​ :There are several testing frameworks available for data testing that you can consider using with Databricks and Pytest:Great Expectations: Great Expectations is an open-source framework that provides a simple way to create and main...

  • 1 kudos
Labels