cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Kamal2
by • New Contributor II
  • 18411 Views
  • 4 replies
  • 6 kudos

Resolved! PDF Parsing in Notebook

I have pdf files stored in azure adls.i want to parse pdf files in pyspark dataframeshow can i do that ?

Data Engineering
Reply
  • 18411 Views
  • 4 replies
  • 6 kudos
View Replies
Latest Reply
Mykola_Melnyk
New Contributor III
  • 6 kudos

PDF Data Source works now on Databricks.Instruction with example: https://stabrise.com/blog/spark-pdf-on-databricks/

  • 6 kudos
3 More Replies
jsaddam28
by • New Contributor III
  • 50227 Views
  • 25 replies
  • 16 kudos

How to import local python file in notebook?

for example I have one.py and two.py in databricks and I want to use one of the module from one.py in two.py. Usually I do this in my local machine by import statement like below two.py__ from one import module1 . . . How to do this in databricks???...

Data Engineering
Reply
  • 50227 Views
  • 25 replies
  • 16 kudos
View Replies
Latest Reply
PabloCSD
Valued Contributor
  • 16 kudos

This alternative worked for us: https://community.databricks.com/t5/data-engineering/is-it-possible-to-import-functions-from-a-module-in-workspace/td-p/5199

  • 16 kudos
24 More Replies
Jfoxyyc
by • Valued Contributor
  • 4404 Views
  • 6 replies
  • 2 kudos

Is there a way to catch the cancel button or the interrupt button in a Databricks notebook?

I'm running oracledb package and it uses sessions. When you cancel a running query it doesn't close the session even if you have a try catch block because a cancel or interrupt issues a kill command on the process. Is there a method to catch the canc...

Data Engineering
Reply
  • 4404 Views
  • 6 replies
  • 2 kudos
View Replies
Latest Reply
gustavo_woiler
New Contributor II
  • 2 kudos

I was having the same issue and I think I was finally able to solve it!When you simply except and capture the KeyboardInterrupt signal and do not raise it, the notebook gets into an endless cycle of "interrupting..." and never does anything.However, ...

  • 2 kudos
5 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 5265 Views
  • 5 replies
  • 5 kudos

Support of running multiple cells at a time in databricks notebook Hi all,Now databricks notebook supports parallel run of commands in a single notebo...

Support of running multiple cells at a time in databricks notebookHi all,Now databricks notebook supports parallel run of commands in a single notebook that will help run ad hoc queries simultaneously without creating a separate notebook.Once you run...

image.png image
Data Engineering
Reply
  • 5265 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
SunilUIIT
New Contributor II
  • 5 kudos

Hi Team,I am observing that the functionality is not working as expected in the Trial workspace of Databricks. Is there a setting that needs to be enabled to allow independent SQL cells in a Databricks notebook to run in parallel, while dependent cel...

  • 5 kudos
4 More Replies
sgannavaram
by • New Contributor III
  • 2904 Views
  • 3 replies
  • 1 kudos

How to connect to IBM MQ from Databricks notebook?

We are trying to connect to IBM MQ and post message to MQ, which eventually consumed by mainframe application.What are the IBM MQ clients .jars / libraries installed in cluster ? if you have any sample code for connectivity that would be helpful.

Data Engineering
Reply
  • 2904 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
none_ranjeet
New Contributor III
  • 1 kudos

Were you able to do this connection other than rest API which have problem in reading Binary messages, Please suggest

  • 1 kudos
2 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 18494 Views
  • 12 replies
  • 12 kudos

Resolved! dbutils or other magic way to get notebook name or cell title inside notebook cell

Not sure it exists but maybe there is some trick to get directly from python code:NotebookNameCellTitlejust working on some logger script shared between notebooks and it could make my life a bit easier

Data Engineering
Reply
  • 18494 Views
  • 12 replies
  • 12 kudos
View Replies
Latest Reply
rtullis
New Contributor II
  • 12 kudos

I got the solution to work in terms of printing the notebook that I was running; however, what if you have notebook A that calls a function that prints the notebook name, and you run notebook B that %runs notebook A?  I get the notebook B's name when...

  • 12 kudos
11 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 1959 Views
  • 1 replies
  • 5 kudos

Notebook cell output results limit increased- 10,000 rows or 2 MB. Hi all, Now, databricks start showing the first 10000 rows instead of 1000 rows.Tha...

Notebook cell output results limit increased- 10,000 rows or 2 MB.Hi all,Now, databricks start showing the first 10000 rows instead of 1000 rows.That will reduce the time of re-execution while working on fewer sizes of data that have rows between 100...

Data Engineering
Reply
  • 1959 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
F_Goudarzi
New Contributor III
  • 5 kudos

Hi Ajay,Is there any way to increase this limit?Thanks, Fatima

  • 5 kudos
elikvar
by • New Contributor III
  • 20723 Views
  • 9 replies
  • 9 kudos

Cluster occasionally fails to launch

I have a daily running notebook that occasionally fails with the error:"Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxx is in un...

Data Engineering
Reply
  • 20723 Views
  • 9 replies
  • 9 kudos
View Replies
Latest Reply
Pavan578
New Contributor II
  • 9 kudos

Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...

  • 9 kudos
8 More Replies
Paddy_chu
by • New Contributor
  • 25815 Views
  • 3 replies
  • 3 kudos

How to restart the kernel on my notebook in databricks?

while installing a python package on my databricks notebook, I kept getting a message saying that: "Note: you may need to restart the kernel using dbutils.library.restartPython() to use updated packages."I've tried restarting my cluster, also detach ...

error message
Data Engineering
Reply
  • 25815 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
johnb1
Contributor
  • 3 kudos

@Evan_MCK Follow-up question:When other notebooks run Python code on the same cluster, will those runs be aborted when dbutils.library.restartPython() is called?

  • 3 kudos
2 More Replies
ekdz__
by • New Contributor III
  • 5744 Views
  • 5 replies
  • 10 kudos

Is there any way to save the notebook in the "Results Only" view?

Hi! I'm looking for a solution to save a notebook in HTML format that has the "Results Only" view (without the executed code). Is there any possibility to do that?Thank you

Data Engineering
Reply
  • 5744 Views
  • 5 replies
  • 10 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 10 kudos

Use option "+New dashboard" in the top menu (picture icon). Add results there (use display() in code to show data), and then you can export the dashboard to HTML.

  • 10 kudos
4 More Replies
del1000
by • New Contributor III
  • 19089 Views
  • 8 replies
  • 3 kudos

Resolved! Is it possible to passthrough job's parameters to variable?

Scenario:I tried to run notebook_primary as a job with same parameters' map. This notebook is orchestrator for notebooks_sec_1, notebooks_sec_2, and notebooks_sec_3 and next. I run them by dbutils.notebook.run(path, timeout, arguments) function.So ho...

Data Engineering
Reply
  • 19089 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
nnalla
New Contributor II
  • 3 kudos

I am using getCurrentBindings(), but it returns an empty dictionary even though I passed parameters. I am running it in a scheduled workflow job

  • 3 kudos
7 More Replies
karolinalbinsso
by • New Contributor II
  • 3155 Views
  • 2 replies
  • 3 kudos

Resolved! How to access the job-Scheduling Date from within the notebook?

I have created a job that contains a notebook that reads a file from Azure Storage. The file-name contains the date of when the file was transferred to the storage. A new file arrives every Monday, and the read-job is scheduled to run every Monday. I...

Data Engineering
Reply
  • 3155 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Hi, I guess the files are in the same directory structure so that you can use cloud files autoloader. It will incrementally read only new files https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loaderSo it will ...

  • 3 kudos
1 More Replies
akihiko
by • New Contributor III
  • 3663 Views
  • 4 replies
  • 1 kudos

Resolved! Attach notebook to cluster via REST API

Is it possible to attach a notebook to cluster and run it via the REST API?The closest approach I have found is to run a notebook, export the results (HTML!) and import it into the workspace again, but this does not allow us to retain the original ex...

Data Engineering
Reply
  • 3663 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
baert23
New Contributor II
  • 1 kudos

I'm looking for a way to programmatically copy a notebook in Databricks using the workspace/export and workspace/import APIs. Once the notebook is copied, I want to automatically attach it to a specific cluster using its cluster ID. The challenge is ...

  • 1 kudos
3 More Replies
yalei
by • New Contributor
  • 5935 Views
  • 1 replies
  • 0 kudos

leaflet not works in notebook(R language)

I saw this notebook: htmlwidgets-azure - Databricks (microsoft.com)However, it is not reproducible. I got a lot errors:there is no package called ‘R.utils’. This is easy to fix, just install the package "R.utils""can not be unloaded". This is not ...

Data Engineering
Reply
  • 5935 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KAdamatzky
New Contributor II
  • 0 kudos

Hi yalei,  Did you have any luck fixing this issue? I am also trying to replicate the htmlwidgets notebook and am running into the same error.Unfortunately, the suggestions provided by Kaniz_Fatma below did not work.

  • 0 kudos
gaurav_khanna
by • New Contributor II
  • 5879 Views
  • 4 replies
  • 3 kudos

Resolved! Notebook is not attaching to a cluster, asks to contact your administrator. Completely stumped. Please help.

Data Engineering
Reply
  • 5879 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
BartRJD
New Contributor II
  • 3 kudos

I am having the same issue (Azure Databricks).I have a running compute cluster analytics-compute-cluster running in Single User access mode.  The Event Log for the cluster says the cluster is running and the "Driver is healthy".I have Manage permissi...

  • 3 kudos
3 More Replies
Labels