Databricks Community

Hi Umesh,If you want to completely ignore the null/empty values then you could simply filter before you do the groupBy, but are you wanting to keep those values?If you want to keep the null values and avoid the skew, you could try splitting the DataF...

generally when u deal with apostrophe u replace the the single quote(') with (''). More about....handling single quotes Antonio

Your schema is tight, but make sure that the conversion to it does not throw an exception. Try with Memory Optimized Nodes, you may be fine. My problem was parsing a lot of data from sequence files containing 10K xml files and saving them as a table...

We came across similar situation we are using spark 1.6.1, we have a daily load process to pull data from oracle and write as parquet files, this works fine for 18 days of data (till 18th run), the problem comes after 19th run where the data frame l...

Got it - how about using a UnionAll? I believe this code snippet does what you'd want:from pyspark.sql import Row array = [Row(value=1), Row(value=2), Row(value=3)] df = sqlContext.createDataFrame(sc.parallelize(array)) array2 = [Row(value=4), Ro...

@dan11 We don't support number in Spark SQL. Try using int, double, float, and your query should be fine. To run SQL in a notebook, just prepend any cell with %sql. %sql select cast(myage as double) as my_integer_age from ages;

I'm facing the same problem, does anybody know how to connect Spark in Ipython notebook? The issue I created, https://github.com/jupyter/notebook/issues/743

Use take()

Analyze is not needed with parquet tables that use the databricks parquet package. That is the default now when you use .saveAsTable(), but if you use a different output format - it's possible that analyze may not work yet.

Looks like the following property is pretty high, which consumes a lot of memory on your executors when you cache the dataset. "spark.storage.memoryFraction:0.9" This could likely be solved by changing the configuration. Take a look at the upstream...

There is no hardcoded limit we just call panda.fromRecords with a collection of fields to instantiate a new Panda Dataframe. The only limit is memory. See http://stackoverflow.com/questions/15455722/pandas-is-there-a-max-size-max-no-of-columns-max-r...

Sorted DataIf your data is sorted using either sort() or ORDER BY, these operations will be deterministic and return either the 1st element using first()/head() or the top-n using head(n)/take(n).show()/show(n) return Unit (void) and will print up to...

How do you set this size ?