What is the difference between spark.sessionState.catalog.listTables vs spark.catalog.listTables

[User16869510359]

I see a significant performance difference when calling spark.sessionState.catalog.list compared to spark.catalog.list. Is that expected?

[User16869510359 · Accepted solution]

spark.sessionState.catalog.listTables is a more lazy implementation.. it does not pull the column details when listing the tables. Hence it's faster. Whereas catalog.listTables will pull the column details as well.

If the database has many Delta tables, it's possible getting the column details will take a long time, whereas just getting the metadata should be faster.