Does spark utilise a temporary stage when writing to snowflake? How does that work?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-03-2022 11:58 PM
Folks , when I want to push data to snowflake I need to use a stage for files before copying data over. However, when I utilise the net.snowflake.spark.snowflake.Utils library and do a spark.write as in...
spark.read.format("csv")
.option("header", "true")
.load("/mydata.data")
.write.format("snowflake")
.options(options).mode("append").option("dbtable", "mytable").save()I can write directly into snowflake without specifying a stage etc.
Just wondering how this works so I can preempt any security problems etc.
Labels:
- Labels:
-
Snowflake Spark Connector
-
Temporary