topic Re: Does spark utilise a temporary stage when writing to snowflake? How does that work? in Data Engineering https://community.databricks.com/t5/data-engineering/does-spark-utilise-a-temporary-stage-when-writing-to-snowflake/m-p/11262#M6268 <P>Yes it uses a temporary stage. should have just looked in snowflake history</P><P><span class="lia-inline-image-display-wrapper" image-alt="image"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1655i90EEC08A07C00096/image-size/large?v=v2&amp;px=999" role="button" title="image" alt="image" /></span></P> Thu, 04 Aug 2022 08:47:37 GMT mick042 2022-08-04T08:47:37Z Does spark utilise a temporary stage when writing to snowflake? How does that work? https://community.databricks.com/t5/data-engineering/does-spark-utilise-a-temporary-stage-when-writing-to-snowflake/m-p/11261#M6267 <P>Folks , when I want to push data to snowflake I need to use a stage for files before copying data over. However, when I utilise the net.snowflake.spark.snowflake.Utils library and do a spark.write as in...</P><P></P><PRE><CODE>spark.read.format("csv") .option("header", "true") .load("/mydata.data") .write.format("snowflake") .options(options).mode("append").option("dbtable", "mytable").save()</CODE></PRE><P>I can write directly into snowflake without specifying a stage etc.</P><P>Just wondering how this works so I can preempt any security problems etc.</P><P></P> Thu, 04 Aug 2022 06:58:42 GMT https://community.databricks.com/t5/data-engineering/does-spark-utilise-a-temporary-stage-when-writing-to-snowflake/m-p/11261#M6267 mick042 2022-08-04T06:58:42Z Re: Does spark utilise a temporary stage when writing to snowflake? How does that work? https://community.databricks.com/t5/data-engineering/does-spark-utilise-a-temporary-stage-when-writing-to-snowflake/m-p/11262#M6268 <P>Yes it uses a temporary stage. should have just looked in snowflake history</P><P><span class="lia-inline-image-display-wrapper" image-alt="image"><img src="https://community.databricks.com/t5/image/serverpage/image-id/1655i90EEC08A07C00096/image-size/large?v=v2&amp;px=999" role="button" title="image" alt="image" /></span></P> Thu, 04 Aug 2022 08:47:37 GMT https://community.databricks.com/t5/data-engineering/does-spark-utilise-a-temporary-stage-when-writing-to-snowflake/m-p/11262#M6268 mick042 2022-08-04T08:47:37Z