cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Kamal2
by • New Contributor II
  • 16593 Views
  • 3 replies
  • 4 kudos

Resolved! PDF Parsing in Notebook

I have pdf files stored in azure adls.i want to parse pdf files in pyspark dataframeshow can i do that ?

Data Engineering
Reply
  • 16593 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Mykola_Melnyk
New Contributor II
  • 4 kudos

Please look to the PDF DataSource for Apache Spark.This project provides a custom data source for the Apache Spark that allows you to read PDF files into the Spark DataFrame. And here notebook with example of usage.df = spark.read.format("pdf") \ ...

  • 4 kudos
2 More Replies
Hubert-Dudek
by • Esteemed Contributor III
  • 17167 Views
  • 12 replies
  • 12 kudos

Resolved! dbutils or other magic way to get notebook name or cell title inside notebook cell

Not sure it exists but maybe there is some trick to get directly from python code:NotebookNameCellTitlejust working on some logger script shared between notebooks and it could make my life a bit easier

Data Engineering
Reply
  • 17167 Views
  • 12 replies
  • 12 kudos
View Replies
Latest Reply
rtullis
New Contributor II
  • 12 kudos

I got the solution to work in terms of printing the notebook that I was running; however, what if you have notebook A that calls a function that prints the notebook name, and you run notebook B that %runs notebook A?  I get the notebook B's name when...

  • 12 kudos
11 More Replies
Ajay-Pandey
by • Esteemed Contributor III
  • 1792 Views
  • 1 replies
  • 5 kudos

Notebook cell output results limit increased- 10,000 rows or 2 MB. Hi all, Now, databricks start showing the first 10000 rows instead of 1000 rows.Tha...

Notebook cell output results limit increased- 10,000 rows or 2 MB.Hi all,Now, databricks start showing the first 10000 rows instead of 1000 rows.That will reduce the time of re-execution while working on fewer sizes of data that have rows between 100...

Data Engineering
Reply
  • 1792 Views
  • 1 replies
  • 5 kudos
View Replies
Latest Reply
F_Goudarzi
New Contributor III
  • 5 kudos

Hi Ajay,Is there any way to increase this limit?Thanks, Fatima

  • 5 kudos
Ajay-Pandey
by • Esteemed Contributor III
  • 4586 Views
  • 4 replies
  • 5 kudos

Support of running multiple cells at a time in databricks notebook Hi all,Now databricks notebook supports parallel run of commands in a single notebo...

Support of running multiple cells at a time in databricks notebookHi all,Now databricks notebook supports parallel run of commands in a single notebook that will help run ad hoc queries simultaneously without creating a separate notebook.Once you run...

image.png image
Data Engineering
Reply
  • 4586 Views
  • 4 replies
  • 5 kudos
View Replies
Latest Reply
SenthilRT
New Contributor III
  • 5 kudos

Can we run parallel cell execution for python (pyspark) cells?

  • 5 kudos
3 More Replies
elikvar
by • New Contributor III
  • 19537 Views
  • 9 replies
  • 9 kudos

Cluster occasionally fails to launch

I have a daily running notebook that occasionally fails with the error:"Run result unavailable: job failed with error message Unexpected failure while waiting for the cluster Some((xxxxxxxxxxxxxxx) )to be readySome(: Cluster xxxxxxxxxxxxxxxx is in un...

Data Engineering
Reply
  • 19537 Views
  • 9 replies
  • 9 kudos
View Replies
Latest Reply
Pavan578
New Contributor II
  • 9 kudos

Cluster 'xxxxxxx' was terminated. Reason: WORKER_SETUP_FAILURE (SERVICE_FAULT). Parameters: databricks_error_message:DBFS Daemomn is not reachable., gcp_error_message:Unable to reach the colocated DBFS Daemon.Can Anyone help me how can we resolve thi...

  • 9 kudos
8 More Replies
Paddy_chu
by • New Contributor
  • 23175 Views
  • 3 replies
  • 3 kudos

How to restart the kernel on my notebook in databricks?

while installing a python package on my databricks notebook, I kept getting a message saying that: "Note: you may need to restart the kernel using dbutils.library.restartPython() to use updated packages."I've tried restarting my cluster, also detach ...

error message
Data Engineering
Reply
  • 23175 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
johnb1
Contributor
  • 3 kudos

@Evan_MCK Follow-up question:When other notebooks run Python code on the same cluster, will those runs be aborted when dbutils.library.restartPython() is called?

  • 3 kudos
2 More Replies
ekdz__
by • New Contributor III
  • 5437 Views
  • 5 replies
  • 10 kudos

Is there any way to save the notebook in the "Results Only" view?

Hi! I'm looking for a solution to save a notebook in HTML format that has the "Results Only" view (without the executed code). Is there any possibility to do that?Thank you

Data Engineering
Reply
  • 5437 Views
  • 5 replies
  • 10 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 10 kudos

Use option "+New dashboard" in the top menu (picture icon). Add results there (use display() in code to show data), and then you can export the dashboard to HTML.

  • 10 kudos
4 More Replies
del1000
by • New Contributor III
  • 18514 Views
  • 8 replies
  • 3 kudos

Resolved! Is it possible to passthrough job's parameters to variable?

Scenario:I tried to run notebook_primary as a job with same parameters' map. This notebook is orchestrator for notebooks_sec_1, notebooks_sec_2, and notebooks_sec_3 and next. I run them by dbutils.notebook.run(path, timeout, arguments) function.So ho...

Data Engineering
Reply
  • 18514 Views
  • 8 replies
  • 3 kudos
View Replies
Latest Reply
nnalla
New Contributor II
  • 3 kudos

I am using getCurrentBindings(), but it returns an empty dictionary even though I passed parameters. I am running it in a scheduled workflow job

  • 3 kudos
7 More Replies
karolinalbinsso
by • New Contributor II
  • 3014 Views
  • 2 replies
  • 3 kudos

Resolved! How to access the job-Scheduling Date from within the notebook?

I have created a job that contains a notebook that reads a file from Azure Storage. The file-name contains the date of when the file was transferred to the storage. A new file arrives every Monday, and the read-job is scheduled to run every Monday. I...

Data Engineering
Reply
  • 3014 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Hi, I guess the files are in the same directory structure so that you can use cloud files autoloader. It will incrementally read only new files https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/auto-loaderSo it will ...

  • 3 kudos
1 More Replies
akihiko
by • New Contributor III
  • 3380 Views
  • 4 replies
  • 1 kudos

Resolved! Attach notebook to cluster via REST API

Is it possible to attach a notebook to cluster and run it via the REST API?The closest approach I have found is to run a notebook, export the results (HTML!) and import it into the workspace again, but this does not allow us to retain the original ex...

Data Engineering
Reply
  • 3380 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
baert23
New Contributor II
  • 1 kudos

I'm looking for a way to programmatically copy a notebook in Databricks using the workspace/export and workspace/import APIs. Once the notebook is copied, I want to automatically attach it to a specific cluster using its cluster ID. The challenge is ...

  • 1 kudos
3 More Replies
yalei
by • New Contributor
  • 5810 Views
  • 1 replies
  • 0 kudos

leaflet not works in notebook(R language)

I saw this notebook: htmlwidgets-azure - Databricks (microsoft.com)However, it is not reproducible. I got a lot errors:there is no package called ‘R.utils’. This is easy to fix, just install the package "R.utils""can not be unloaded". This is not ...

Data Engineering
Reply
  • 5810 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
KAdamatzky
New Contributor II
  • 0 kudos

Hi yalei,  Did you have any luck fixing this issue? I am also trying to replicate the htmlwidgets notebook and am running into the same error.Unfortunately, the suggestions provided by Kaniz_Fatma below did not work.

  • 0 kudos
gaurav_khanna
by • New Contributor II
  • 5367 Views
  • 4 replies
  • 3 kudos

Resolved! Notebook is not attaching to a cluster, asks to contact your administrator. Completely stumped. Please help.

Data Engineering
Reply
  • 5367 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
BartRJD
New Contributor II
  • 3 kudos

I am having the same issue (Azure Databricks).I have a running compute cluster analytics-compute-cluster running in Single User access mode.  The Event Log for the cluster says the cluster is running and the "Driver is healthy".I have Manage permissi...

  • 3 kudos
3 More Replies
Jfoxyyc
by • Valued Contributor
  • 4009 Views
  • 4 replies
  • 0 kudos

Is there a way to catch the cancel button or the interrupt button in a Databricks notebook?

I'm running oracledb package and it uses sessions. When you cancel a running query it doesn't close the session even if you have a try catch block because a cancel or interrupt issues a kill command on the process. Is there a method to catch the canc...

Data Engineering
Reply
  • 4009 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
jonathan-dufaul
Valued Contributor
  • 0 kudos

I'm having the same issue and this has been frustrating as heck.

  • 0 kudos
3 More Replies
sgannavaram
by • New Contributor III
  • 2719 Views
  • 2 replies
  • 1 kudos

How to connect to IBM MQ from Databricks notebook?

We are trying to connect to IBM MQ and post message to MQ, which eventually consumed by mainframe application.What are the IBM MQ clients .jars / libraries installed in cluster ? if you have any sample code for connectivity that would be helpful.

Data Engineering
Reply
  • 2719 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Saleem
New Contributor II
  • 1 kudos

Kindly update if you are able to connect to MQ from Databricks. I am working on same but no luck as I’m unable to install pymqi library on the cluster as its showing error as MQ library could not be found

  • 1 kudos
1 More Replies
shiv4050
by • New Contributor
  • 3434 Views
  • 4 replies
  • 0 kudos

Execute databricks notebook form a python source code.

Hello,I 'm trying to execute databricks notebook form a python source code but getting error.source code below------------------from databricks_api import DatabricksAPI   # Create a Databricks API client api = DatabricksAPI(host='databrick_host', tok...

Data Engineering
Reply
  • 3434 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
sewl
New Contributor II
  • 0 kudos

The error you are encountering indicates that there is an issue with establishing a connection to the Databricks host specified in your code. Specifically, the error message "getaddrinfo failed" suggests that the hostname or IP address you provided f...

  • 0 kudos
3 More Replies
Labels