Hi @faruk 

I understand the situation, but you can show this reply to your security team and try to get a view created. Once you have that access, you can use a recursive function to get the lineage of lineage in memory.

Here is the exact logic I used to build it (You need to change the code a bit):

def _build_graph(self):
        print("Building in-memory dependency graph...")
        
        query = """
            SELECT DISTINCT
                target_table_full_name AS object,
                target_type AS object_type,
                source_table_full_name AS depends_on_object,
                source_type AS depends_on_object_type
            FROM 
                system.access.table_lineage
            WHERE 
                source_type IN ('TABLE', 'VIEW') 
                AND target_type IN ('TABLE', 'VIEW')
                AND target_table_full_name IS NOT NULL
                AND source_table_full_name IS NOT NULL
        """
        df_lineage = self.spark.sql(query).collect()
        
        for row in df_lineage:
            obj = row['object'].lower()
            obj_type = row['object_type']
            
            dep = row['depends_on_object'].lower()
            dep_type = row['depends_on_object_type']
            
            # Store valid names
            self.all_known_tables.add(obj)
            self.all_known_tables.add(dep)
            
            # Store their types
            self.object_types[obj] = obj_type
            self.object_types[dep] = dep_type
            
            # Build relationships
            if obj not in self.dependency_graph:
                self.dependency_graph[obj] = set()
            self.dependency_graph[obj].add(dep)
            
        print(f"Graph built successfully. Tracking {len(self.dependency_graph)} target objects.")

The best part about doing it this way is that it solves your deleted data problem. You just cross-reference that all_known_tables set against the active system.information_schema.tables. If an object doesn't exist there anymore, you know it was deleted and can safely drop it from your dictionary.