Databricks Community

rlgarris · 02-10-2016

A cartesian product is a common operation to get the cross product of two tables. For example, say you have a list of customers and a list of your product catalog and want to get the cross product of all customer - product combinations. Cartesian pr...

rlgarris · 12-02-2015

Using sparkcsv to write data to dbfs, which I plan to move to my laptop via standard s3 copy commands. The default for spark csv is to write output into partitions. I can force it to a single partition, but would really like to know if there is a ge...

rlgarris · 12-27-2016

Hi - KNN is notoriously hard to parallelize in Spark because KNN is a "lazy learner" and the model itself is the entire dataset. Most single machine implementations rely on KD Trees or Ball Trees to store the entire dataset in the RAM of a single ma...

rlgarris · 09-09-2016

Hi nmud19, What error did you get? Can you paste a stack trace? You may need to make the path absolute and set recursive to true. dbutils.fs.rm("/mnt/temp", true)

rlgarris · 02-17-2016

Hi @cfregly One correction. registerTempTable does not create a "cached" in-memory table, but rather an alias or a reference to the DataFrame. It's akin to a pointer in C/C++ or a reference in Java. Every time you query the temp table you are actual...

rlgarris · 02-09-2016

Hi, You can also using Windowing functions https://databricks.com/blog/2015/07/15/introducing-window-functions-in-spark-sql.html'; percent_rank at 0.25, 0.50 and 0.75 will give you want you are looking for.

rlgarris · 12-02-2015

Thanks Richard. That is useful for single files. I'll add it to our local docs. I ended up writing a shell script that downloads all parts and merges them locally, so that can remain an option for people with larger files.

Databricks Community

User Stats

User Activity

How do I get a cartesian product of a huge dataset?

How do I create a single CSV file from multiple partitions in Databricks / Spark?

Re: KNN classifier on Spark

Re: how to delete a folder in databricks mnt?

Re: What is the difference between registerTempTable() and saveAsTable()?

Re: calculate median and inter quartile range on spark dataframe

Re: How do I create a single CSV file from multiple partitions in Databricks / Spark?