https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-spark-sessionstate-catalog/m-p/21338#M14538 <P>I see a significant performance difference when calling spark.sessionState.catalog.list compared to spark.catalog.list. Is that expected? </P> Wed, 23 Jun 2021 19:25:51 GMT brickster_2018 2021-06-23T19:25:51Z https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-spark-sessionstate-catalog/m-p/21338#M14538 <P>I see a significant performance difference when calling spark.sessionState.catalog.list compared to spark.catalog.list. Is that expected? </P> Wed, 23 Jun 2021 19:25:51 GMT https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-spark-sessionstate-catalog/m-p/21338#M14538 brickster_2018 2021-06-23T19:25:51Z https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-spark-sessionstate-catalog/m-p/21339#M14539 <P>spark.sessionState.catalog.listTables is a more lazy implementation.. it does not pull the column details when listing the tables. Hence it's faster. Whereas catalog.listTables will pull the column details as well. </P><P>If the database has many Delta tables, it's possible getting the column details will take a long time, whereas just getting the metadata should be faster. </P> Wed, 23 Jun 2021 19:29:28 GMT https://community.databricks.com/t5/data-engineering/what-is-the-difference-between-spark-sessionstate-catalog/m-p/21339#M14539 brickster_2018 2021-06-23T19:29:28Z