https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3898#M778 <P>writing without the check could lead to corrupt data, so I doubt this is possible.</P> Tue, 30 May 2023 09:14:49 GMT -werners- 2023-05-30T09:14:49Z https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3897#M777 <P>After spark finishes writing the dataframe to S3, it seems like it checks the validity of the files it wrote with:&nbsp;`getFileStatus` that is&nbsp;`HeadObject`</P><P>&nbsp;behind the scenes.</P><P>What if I'm only granted write and list objects permissions but not GetObject? Is there any way instructing pyspark on databricks to not do this validity test after a successful write?</P> Mon, 29 May 2023 23:16:47 GMT https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3897#M777 gdoron 2023-05-29T23:16:47Z https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3898#M778 <P>writing without the check could lead to corrupt data, so I doubt this is possible.</P> Tue, 30 May 2023 09:14:49 GMT https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3898#M778 -werners- 2023-05-30T09:14:49Z https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3899#M779 <P>It is not possible in my opinion.</P> Tue, 30 May 2023 11:00:29 GMT https://community.databricks.com/t5/data-engineering/using-pyspark-can-i-write-to-an-s3-path-i-don-t-have-getobject/m-p/3899#M779 Lakshay 2023-05-30T11:00:29Z