running notebook job from remote github repository fails, but do not fail on python script type

NielsMH
New Contributor III

‎10-18-2023 08:16 AM

Hi all

I am trying to run a notebook from a remote repository, but the job fails. 

I setup the job as follows:

job-setup.png

my project structure is as such:

folder_structure.png

but the output i get is like such:

job_output.png

 

The thing is if i set the job type to "Python Script" i dont encounter this problem. Then the job will carry on as expected with no errors. However, i need the poc_template_job_script.py to be read as a notebook. So somehow i think the job is not recognizing my project notebook (poc_template_job_script.py).

Anyone has some input to how i can solve this issue? I have not found a similar issue anywhere.

Thanks a lot in advance 🙂

0 Kudos

karthik_p
Databricks Partner

‎10-18-2023 09:32 AM

@NielsMH if you want to run your jobs based o job name, please use new preview service that databricks released which are DAB format. there you can run your job based on your job name.

remote repo in the sense, are you using github actions or api, looks notebook format is not being detected untill you specify type of format (in u r case py)

karthik.p
0 Kudos

NielsMH
New Contributor III
In response to karthik_p

‎10-19-2023 05:53 AM

Hi Karthik_p thanks for the reply

If i specify the extension poc_template_job_script.py, I get the same error. In the setup it asks to omit the extension which results in my initial post. And i agree that it should work if i specified the extension if you look at the documentation:

Supported notebook formats

Databricks can import and export notebooks in the following formats:

  • Source file: A file containing only source code statements with the extension .scala, .py, .sql, or .r.

But somehow it still does not recognize poc_template_job_script.py as a databricks notebook.

I also tried to add: "# Databricks notebook source" in the top of the file as specified in the comments also:

Import a file and convert it to a notebook

You can convert Python, SQL, Scala, and R scripts to single-cell notebooks by adding a comment to the first cell of the file:

# Databricks notebook source