Im trying to get the lineage graph to work in Unity catalog, however nothing seems to appear even though I followed the docs. I did the following steps

1. Created a Unity metastore and attached the workspace to that metastore.

2. Created a Single user Spark 11.2 cluster within the linked workspace with the following option:

spark.databricks.dataLineage.enabled true

3. Created a catalog lineage_data and a schema within that called lineagedemo managed by the Unity metastore.

4. Executed the following script with the previously created Spark Cluster (lineage demo: https://docs.databricks.com/data-governance/unity-catalog/data-lineage.html 😞

CREATE TABLE IF NOT EXISTS
  lineage_data.lineagedemo.menu (
    recipe_id INT,
    app string,
    main string,
    dessert string
  );
 
INSERT INTO lineage_data.lineagedemo.menu
    (recipe_id, app, main, dessert)
VALUES
    (1,"Ceviche", "Tacos", "Flan"),
    (2,"Tomato Soup", "Souffle", "Cree Brulee"),
    (3,"Chips","Grilled Cheese","Cheesecake");
 
CREATE TABLE
  lineage_data.lineagedemo.dinner
AS SELECT
  recipe_id, concat(app," + ", main," + ",dessert)
AS
  full_menu
FROM
  lineage_data.lineagedemo.menu
 

5. Go to the data explorer and open the lineage tab for the table dinner.

No lineage is visible.. What Am i doing wrong? Where can I find more logging regarding the lineage extraction?