Im trying to get the lineage graph to work in Unity catalog, however nothing seems to appear even though I followed the docs. I did the following steps
1. Created a Unity metastore and attached the workspace to that metastore.
2. Created a Single user Spark 11.2 cluster within the linked workspace with the following option:
spark.databricks.dataLineage.enabled true
3. Created a catalog lineage_data and a schema within that called lineagedemo managed by the Unity metastore.
4. Executed the following script with the previously created Spark Cluster (lineage demo: https://docs.databricks.com/data-governance/unity-catalog/data-lineage.html ๐
CREATE TABLE IF NOT EXISTS
lineage_data.lineagedemo.menu (
recipe_id INT,
app string,
main string,
dessert string
);
INSERT INTO lineage_data.lineagedemo.menu
(recipe_id, app, main, dessert)
VALUES
(1,"Ceviche", "Tacos", "Flan"),
(2,"Tomato Soup", "Souffle", "Cree Brulee"),
(3,"Chips","Grilled Cheese","Cheesecake");
CREATE TABLE
lineage_data.lineagedemo.dinner
AS SELECT
recipe_id, concat(app," + ", main," + ",dessert)
AS
full_menu
FROM
lineage_data.lineagedemo.menu
5. Go to the data explorer and open the lineage tab for the table dinner.
No lineage is visible.. What Am i doing wrong? Where can I find more logging regarding the lineage extraction?
