cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
mick042
New Contributor III
since ‎06-09-2022
‎06-26-2023

User Stats

  • 6 Posts
  • 1 Solutions
  • 0 Kudos given
  • 3 Kudos received

User Activity

Does spark utilise a temporary stage when writing to snowflake? How does that work?

08-03-2022
Folks , when I want to push data to snowflake I need to use a stage for files before copying data over. However, when I utilise the net.snowflake.spark.snowflake.Utils library and do a spark.write as in...spark.read.format("csv") .option("header", ...

Optimal approach when using external script/executable for processing data

06-14-2022
I need to process a number of files where I manipulate file text utilising an external executable that operates on stdin/stdout. I am quite new to spark. What I am attempting is to use rdd.pipe as in the followingexe_path = " /usr/local/bin/external...

Spark Executor Rdd.Pipe call not finding file location that exists in Sparkfiles.get()

06-10-2022
In a Databricks notebook, I need to run text files (stdin, stdout) through a function from an external library. I have used sparkContext.AddFiles({external_library_name}) to add the external library so that it is available to all executors.when I run...
Kudos from