Selective column loader unity catalog

vk217 — Wed, 27 Sep 2023 12:27:05 GMT

I am loading a table into a data frame using

df = spark.table(table_name)

Is there a way to load only the required columns? The table has more than 50+ columns and I only need a handful of column.

Re: Selective column loader unity catalog

daniel_sahal — Thu, 28 Sep 2023 05:39:55 GMT

@vk217 Simply just use select function, ex.

df = spark.read.table(table_name).select("col1", "col2", "col3")