- 13534 Views
- 7 replies
- 0 kudos
I have file a.csv or a.parquet while creating data frame reading we can explictly define schema with struct type. instead of write the schema in the notebook want to create schema lets say for all my csv i have one schema like csv_schema and stored ...
- 13534 Views
- 7 replies
- 0 kudos
Latest Reply
@shyampsr big thanks, was searching for the solution almost 3 hours _https://luckycanadian.com/
6 More Replies
- 10983 Views
- 3 replies
- 1 kudos
I am creating dataframe using SQL in which all the underline tables are actually tempview based on dataframes. I am getting below error everytime. Can anyone help me to uderstand the issue here. Thanks in advance.An error occurred while calling o183....
- 10983 Views
- 3 replies
- 1 kudos
Latest Reply
I also encountered the same problem...It's weird that I can do the query but not the count.
2 More Replies
- 11349 Views
- 2 replies
- 0 kudos
I am new to Spark and just started an online pyspark tutorial. I uploaded the json data in DataBrick and wrote the commands as follows:
df = sqlContext.sql("SELECT * FROM people_json")
df.printSchema()
from pyspark.sql.types import *
data_schema =...
- 11349 Views
- 2 replies
- 0 kudos
- 9374 Views
- 2 replies
- 0 kudos
I have the following sparkdataframe :
agent_id/ payment_amount
a /1000
b /1100
a /1100
a /1200
b /1200
b /1250
a /10000
b /9000
my desire output would be something like
<code>agen_id 95_quantile
a whatever is95 quantile for a...
- 9374 Views
- 2 replies
- 0 kudos
Latest Reply
For those of you who haven't run into this SO thread http://stackoverflow.com/questions/39633614/calculate-quantile-on-grouped-data-in-spark-dataframe, it's pointed out there that one work-around is to use HIVE UDF "percentile_approx". Please see th...
1 More Replies
- 12421 Views
- 2 replies
- 0 kudos
Hey all,
I am trying to filter on a string but the string has a single quote - how do I escape the string in Scala?
I have tried an old version of StringEscapeUtils but no luck.
Sorry if a silly question - new to Scala.import org.apache.commons.lan...
- 12421 Views
- 2 replies
- 0 kudos
Latest Reply
generally when u deal with apostrophe u replace the the single quote(') with ('').
More about....handling single quotes
Antonio
1 More Replies
by
Sri1
• New Contributor II
- 12083 Views
- 5 replies
- 0 kudos
Hi,
My requirement is I need to create a Spark In-memory table (Not pushing hive table into memory) insert data into it and finally write that back to Hive table.
Idea here is to avoid the disk IO while writing into Target Hive table. There are lot ...
- 12083 Views
- 5 replies
- 0 kudos
Latest Reply
Got it - how about using a UnionAll? I believe this code snippet does what you'd want:from pyspark.sql import Row
array = [Row(value=1), Row(value=2), Row(value=3)] df = sqlContext.createDataFrame(sc.parallelize(array))
array2 = [Row(value=4), Ro...
4 More Replies
