cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Notebook ID in DESCRIBE HISTORY not showing

lumen
New Contributor II

We've recently installed Databricks 14.3 LTS with Unity Catalog and for some reason that is escaping me the Notebook Id is not showing up when I execute the DESCRIBE HISTORY SQL command. Example below for table 

test_catalog.lineagedemo.lm_lineage_test. 

Can you help? What's strange to me is that in the history table everything else is populated, except the Notebook. Could this be some configuration that I'm missing ?

Image 23-05-2025 at 14.18.png

 

3 REPLIES 3

RameshRetnasamy
Contributor

Hi @lumen, Generally, notebook Id will be populated if the version was created via notebook. I suspect these versions are created by other means? 

Here is an example where I created first 2 versions via a notebook and the 3rd version created via SQL Editor. As  expected notebook id appears for the first 2 versions, but not for the 3rd one.

Screenshot 2025-05-23 at 15.06.58.png

 

lumen
New Contributor II

Hi @RameshRetnasamy first off thank you so much for taking the time to reply to my question. In my case they were indeed created via Notebooks, but I'll re-evaluate on my end as I might've missed something. If the issue persists, I'll re-assert the question.

Cheers 

lumen
New Contributor II

Yeah in my case I'm starting to think this could be an installation thing, or a unset property. So I created the following sample code and got the below history. What could this be? I never had this problem before.

spark.sql("CREATE TABLE IF NOT EXISTS enterprisedata.hr.employees(name STRING, salary INT)")
spark.sql("DROP TABLE IF EXISTS enterprisedata.hr.employees_no_salary")
spark.sql("INSERT INTO enterprisedata.hr.employees VALUES ('Todd',10000)")

## READ TABLE IN UC
emp = spark.read.table("enterprisedata.hr.employees")

## RENAME A COLLUMN
emp = emp.withColumnRenamed("name", "name_employee" )

## DROP COLUMNS
emp = emp.drop("salary")

## WRITE
emp.write.mode("overwrite").saveAsTable("enterprisedata.hr.employees_no_salary")

Image 23-05-2025 at 19.08.pngImage 23-05-2025 at 19.09.png

 

 

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now