topic What is the most efficient way to read in a partitioned parquet file with pyspark? in Data Engineering https://community.databricks.com/t5/data-engineering/what-is-the-most-efficient-way-to-read-in-a-partitioned-parquet/m-p/20738#M14036 <P>I work with parquet files stored in AWS S3 buckets. They are multiple TB in size and partitioned by a numeric column containing integer values between 1 and 200, call it my_partition. I read in and perform compute actions on this data in Databricks with autoscaling turned off.</P> Thu, 24 Jun 2021 15:09:20 GMT User16790091296 2021-06-24T15:09:20Z What is the most efficient way to read in a partitioned parquet file with pyspark? https://community.databricks.com/t5/data-engineering/what-is-the-most-efficient-way-to-read-in-a-partitioned-parquet/m-p/20738#M14036 <P>I work with parquet files stored in AWS S3 buckets. They are multiple TB in size and partitioned by a numeric column containing integer values between 1 and 200, call it my_partition. I read in and perform compute actions on this data in Databricks with autoscaling turned off.</P> Thu, 24 Jun 2021 15:09:20 GMT https://community.databricks.com/t5/data-engineering/what-is-the-most-efficient-way-to-read-in-a-partitioned-parquet/m-p/20738#M14036 User16790091296 2021-06-24T15:09:20Z