- 1660 Views
- 1 replies
- 1 kudos
How to make recursive calls to python/pandas UDF? For example, unzipping arbitrarily nested zip files.
There are files that are zip files and have many zip files within them, many levels. How do you read/parse the content?
- 1660 Views
- 1 replies
- 1 kudos
- 1 kudos
'tail-recurse' is a python API that can help.
- 1 kudos
- 926 Views
- 0 replies
- 5 kudos
Some Tips & Tricks for Optimizing costs and performance (Clusters and Ganglia):Â [Note: This list is not exhaustive]Â Leverage the DataFrame or Spar...
Some Tips & Tricks for Optimizing costs and performance (Clusters and Ganglia):[Note: This list is not exhaustive]Leverage the DataFrame or SparkSQL API’s first. They use the same execution process resulting in parity in performance but they also com...
- 926 Views
- 0 replies
- 5 kudos
- 11161 Views
- 7 replies
- 0 kudos
Resolved! How do I register a UDF that returns an array of tuples in scala/spark?
I'm relatively new to Scala. In the past, I was able to do the following python: def foo(p1, p2): import datetime as dt dt.datetime(2014, 4, 17, 12, 34) result = [ (1, "1", 1.1, dt.datetime(2014, 4, 17, 1, 0)), (2, "2", 2...
- 11161 Views
- 7 replies
- 0 kudos
- 0 kudos
Hello, Just in case, here is an example for proposed solution above: import org.apache.spark.sql.functions._ import org.apache.spark.sql.expressions._ import org.apache.spark.sql.types._ val data = Seq(("A", Seq((3,4),(5,6),(7,10))), ("B", Seq((-1,...
- 0 kudos