- 18947 Views
- 5 replies
- 0 kudos
Resolved! Pyspark DataFrame: Converting one column from string to float/double
Pyspark 1.6: DataFrame: Converting one column from string to float/double I have two columns in a dataframe both of which are loaded as string. DF = rawdata.select('house name', 'price') I want to convert DF.price to float. DF = rawdata.select('hous...
- 18947 Views
- 5 replies
- 0 kudos
- 0 kudos
Slightly simpler: df_num = df.select(df.employment.cast("float"), df.education.cast("float"), df.health.cast("float")) This works with multiple columns, three shown here.
- 0 kudos
- 9361 Views
- 3 replies
- 0 kudos
How to set size of Parquet output files ?
Hi I'm using Parquet for format to store Raw Data. Actually the part file are stored on S3 I would like to control the file size of each parquet part file. I try this sqlContext.setConf("spark.parquet.block.size", SIZE.toString) sqlContext.setCon...
- 9361 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi All can anyone tell me what is the default Raw Group size while writing via SparkSql
- 0 kudos
- 10118 Views
- 2 replies
- 0 kudos
how to Calculate quantile on grouped data in spark Dataframe
I have the following sparkdataframe : agent_id/ payment_amount a /1000 b /1100 a /1100 a /1200 b /1200 b /1250 a /10000 b /9000 my desire output would be something like <code>agen_id 95_quantile a whatever is95 quantile for a...
- 10118 Views
- 2 replies
- 0 kudos
- 0 kudos
For those of you who haven't run into this SO thread http://stackoverflow.com/questions/39633614/calculate-quantile-on-grouped-data-in-spark-dataframe, it's pointed out there that one work-around is to use HIVE UDF "percentile_approx". Please see th...
- 0 kudos
- 7548 Views
- 2 replies
- 0 kudos
How to extract year and week number from a columns in a sparkDataFrame?
I have the following sparkdataframe : sale_id/ created_at 1 /2016-05-28T05:53:31.042Z 2 /2016-05-30T12:50:58.184Z 3/ 2016-05-23T10:22:18.858Z 4 /2016-05-27T09:20:15.158Z 5 /2016-05-21T08:30:17.337Z 6 /2016-05-28T07:41:14.361Z i need t add a year-wee...
- 7548 Views
- 2 replies
- 0 kudos
- 16684 Views
- 6 replies
- 0 kudos
Two stage join fails with java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary
Sometimes the error is part of "org.apache.spark.SparkException: Exception thrown in awaitResult:". The error source is the step, where we extract the second time the rows, where the data is updated. We can count the rows, but we cannot display or w...
- 16684 Views
- 6 replies
- 0 kudos
- 0 kudos
Thanks Lleido. I eventually found I had changed the schema of a partitioned DataFrame that I had made inadvertently where I narrowed a column's type from a long to an integer. While rather obvious cause of the problem in hindsight it was terribly di...
- 0 kudos
- 5169 Views
- 2 replies
- 0 kudos
Resolved! agg function not working for multiple aggregations
Data has 2 columns: |requestDate|requestDuration| | 2015-06-17| 104| Here is the code: avgSaveTimesByDate = gridSaves.groupBy(gridSaves.requestDate).agg({"requestDuration": "min", "requestDuration": "max","requestDuration": "avg"}) avgSaveTimesBy...
- 5169 Views
- 2 replies
- 0 kudos
- 0 kudos
My guess is that the reason this may not work is the fact that the dictionary input does not have unique keys. With this syntax, column-names are keys and if you have two or more aggregation for the same column, some internal loops may forget the no...
- 0 kudos
- 4949 Views
- 2 replies
- 0 kudos
Resolved! Select dataframe columns from a sequence of string
Is there a simple way to select columns from a dataframe with a sequence of string? Something like val colNames = Seq("c1", "c2") df.select(colNames)
- 4949 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks. I needed to modify the final lines. val df_new = df.select(column_names_col:_*) df_new.show() Edward
- 0 kudos
- 6552 Views
- 3 replies
- 0 kudos
How to calculate Percentile of column in a DataFrame in spark?
I am trying to calculate percentile of a column in a DataFrame? I cant find any percentile_approx function in Spark aggregation functions. For e.g. in Hive we have percentile_approx and we can use it in the following way hiveContext.sql("select per...
- 6552 Views
- 3 replies
- 0 kudos
- 0 kudos
You can try using df.registerTempTable("tmp_tbl") val newDF = sql(/ do something with tmp_tbl /)// and continue using newDF Learn More
- 0 kudos
- 15276 Views
- 5 replies
- 2 kudos
Markup in Databricks Notebook
Do Databricks Scala Notebooks support any sort of markup/markdown?
- 15276 Views
- 5 replies
- 2 kudos
- 7062 Views
- 3 replies
- 0 kudos
- 7062 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello Pls help (Not an Answer), How do I download complete csv (>1000) result file in FileStore unto my laptop? I was trying to follow this instruction set SQL tutorial (Download All SQL - scala)
- 0 kudos
- 11960 Views
- 1 replies
- 0 kudos
How can i read parquet file compressed by snappy?
Hi All, I wanted to read parqet file compressed by snappy into Spark RDD input file name is: part-m-00000.snappy.parquet i have used sqlContext.setConf("spark.sql.parquet.compression.codec.", "snappy") val inputRDD=sqlContext.parqetFile(args(0)) whe...
- 11960 Views
- 1 replies
- 0 kudos
- 0 kudos
Have you tried sqlContext.read.parquet("/filePath/") ?
- 0 kudos
- 19063 Views
- 5 replies
- 0 kudos
Resolved! Writing DataFrame to PostgreSQL via JDBC extremely slow (Spark 1.6.1)
Hi there,I'm just getting started with Spark and I've got a moderately sized DataFrame created from collating CSVs in S3 (88 columns, 860k rows) that seems to be taking an unreasonable amount of time to insert (using SaveMode.Append) into Postgres. I...
- 19063 Views
- 5 replies
- 0 kudos
- 0 kudos
In case anyone was curious how I worked around this, I ended up dropping down to Postgres JDBC and using CopyManager to COPY rows in directly from Spark: https://gist.github.com/longcao/bb61f1798ccbbfa4a0d7b76e49982f84
- 0 kudos
- 12315 Views
- 3 replies
- 0 kudos
How to avoid empty/null keys in DataFrame groupby?
Hi I have Spark job which does group by and I cant avoid it because of my use case. I have large dataset around 1 TB which I need to process/update in DataFrame. Now my jobs shuffles huge data and slows things because of shuffling and groupby. One r...
- 12315 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Umesh,If you want to completely ignore the null/empty values then you could simply filter before you do the groupBy, but are you wanting to keep those values?If you want to keep the null values and avoid the skew, you could try splitting the DataF...
- 0 kudos
- 14065 Views
- 2 replies
- 0 kudos
How do I escape a query string in Spark SQL?
Hey all, I am trying to filter on a string but the string has a single quote - how do I escape the string in Scala? I have tried an old version of StringEscapeUtils but no luck. Sorry if a silly question - new to Scala.import org.apache.commons.lan...
- 14065 Views
- 2 replies
- 0 kudos
- 0 kudos
generally when u deal with apostrophe u replace the the single quote(') with (''). More about....handling single quotes Antonio
- 0 kudos
- 28148 Views
- 12 replies
- 0 kudos
Why do I get 'java.io.IOException: File already exists' for saveAsTable with Overwrite mode?
I have a fairly small, simple DataFrame, month:month.schema org.apache.spark.sql.types.StructType = StructType(StructField(month,DateType,true), StructField(real_month,TimestampType,true), StructField(month_millis,LongType,true))The month Dataframe i...
- 28148 Views
- 12 replies
- 0 kudos
- 0 kudos
Your schema is tight, but make sure that the conversion to it does not throw an exception. Try with Memory Optimized Nodes, you may be fine. My problem was parsing a lot of data from sequence files containing 10K xml files and saving them as a table...
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 442 | |
| 349 | |
| 287 |
