โ09-27-2023 05:27 AM
I am loading a table into a data frame using
df = spark.table(table_name)
Is there a way to load only the required columns? The table has more than 50+ columns and I only need a handful of column.
โ09-27-2023 10:39 PM
@vk217 Simply just use select function, ex.
df = spark.read.table(table_name).select("col1", "col2", "col3")
View solution in original post
never-displayed
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!