cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Programmatically create Databricks Notebook

wesg2
New Contributor

I am creating a databricks notebook via string concats (sample below)

Notebook_Head = """# Databricks notebook source
# from pyspark.sql.types import StringType
# from pyspark.sql.functions import split

# COMMAND ----------
"""
Full_NB = Notebook_Head + Mid + Tail
 
I am using dbutils.fs.put() to write to a storage location.
When I then try to import that file as a NB in databricks it displays as one large text file, NOT as a Databricks Notebook. 
Line 1 = # Databricks notebook source
 
The notebook does not break up into command cells even though I have the proper # COMMAND ----------
1 REPLY 1

filipniziol
Contributor

Hi @wesg2 ,

One needs to be very precise when building this.

The below code WORKS:

# Define the content of the .py file with cell separators (Works!)
notebook_content = """# Databricks notebook source
# This is the header of the notebook
# You can add imports or setup code here

# COMMAND ----------
# This is the first code cell
print('Hello from the first cell!')

# COMMAND ----------
# This is the second code cell
print('This is another cell.')
"""

# Define the output path (update the path as needed)
output_path = "<your path>/notebook_script.py"

# Save the notebook content to the .py file
dbutils.fs.put(output_path, notebook_content, overwrite=True)

# Inform the user of the location of the saved file
print(f"Notebook saved as .py file at: {output_path}")

The below code does NOT work (extra new line character):

# Define the content of the .py file with cell separators (New Line character - does not work!)
notebook_content = """
# Databricks notebook source
# This is the header of the notebook
# You can add imports or setup code here

# COMMAND ----------
# This is the first code cell
print('Hello from the first cell!')

# COMMAND ----------
# This is the second code cell
print('This is another cell.')
"""

# Define the output path (update the path as needed)
output_path = "<your path>/notebook_script.py"

# Save the notebook content to the .py file
dbutils.fs.put(output_path, notebook_content, overwrite=True)

# Inform the user of the location of the saved file
print(f"Notebook saved as .py file at: {output_path}")

However, I have copied your Notebook_Head, saved this in ADLS as .py file and then imported back to Databricks and it also works. 

Have you tried with just Notebook_Head? If yes, do you followed the same steps as I or do you do it differently?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group