- 12466 Views
- 12 replies
- 0 kudos
Resolved! Custom line separator
I see that https://github.com/apache/spark/pull/18581 will enable defining custom Line Separators for many sources, including CSV. Apart from waiting on this PR to make it into the main Databricks runtime, is there any other alternative to support d...
- 12466 Views
- 12 replies
- 0 kudos
- 0 kudos
You can use newAPIHadoopFile SCALA import org.apache.hadoop.io.LongWritable import org.apache.hadoop.io.Text import org.apache.hadoop.conf.Configuration import org.apache.hadoop.mapreduce.lib.input.TextInputFormat val conf = new Configuration conf.s...
- 0 kudos
- 18405 Views
- 3 replies
- 0 kudos
Resolved! I've set the partition mode to nonstrict in hive but spark is not seeing it
I've got a table I want to add some data to and it's partitoned. I want to use dynamic partitioning but I get this error org.apache.spark.SparkException: Dynamic partition strict mode requires at least one static partition column. To turn this off ...
- 18405 Views
- 3 replies
- 0 kudos
- 0 kudos
I got it working. This was exactly what I needed. Thank you @Peyman Mohajerian​
- 0 kudos
- 11286 Views
- 0 replies
- 0 kudos
TypeError: Column is not iterable when using more than one columns in withColumn()
I am trying to find quarter start date from a date column. I get the expected result when i write it using selectExpr() but when i add the same logic in .withColumn() i get TypeError: Column is not iterableI am using a workaround as follows workarou...
- 11286 Views
- 0 replies
- 0 kudos
- 6452 Views
- 1 replies
- 0 kudos
I want to split a dataframe with date range 1 week, with each week data in different column.
DF Q Date(yyyy-mm-dd) q1 2017-10-01 q2 2017-10-03 q1 2017-10-09 q3 2017-10-06 q2 2017-10-01 q1 2017-10-13 Q1 2017-10-02 Q3 2017-10-21 Q4 2017-10-17 Q5 2017-10-20 Q4 2017-10-31 Q2 2017-10-27 Q5 2017-10-01 Dataframe: ...
- 6452 Views
- 1 replies
- 0 kudos
- 0 kudos
It should just be a matter of applying the correct set of transformations:You can start by adding the week-of-year to each record with the command pyspark.sql.functions.weekofyear(..) and name it something like weekOfYear. See https://spark.apache.or...
- 0 kudos
- 36895 Views
- 0 replies
- 2 kudos
Requesting support for "SELECT TOP n from Table"
In notebook, It looks like if I need to select top N rows, I can rely on "LIMIT" keyword. It would be nice if you can support "TOP" as well The current approach to select 10 rows: select * from table1 LIMIT 10 Requesting TOP support: SELECT TOP 10 *...
- 36895 Views
- 0 replies
- 2 kudos
- 19140 Views
- 3 replies
- 1 kudos
Resolved! How to parse a file with newline character, escaped with \ and not quoted
Hi! I am facing an issue when reading and parsing a CSV file. Some records have a newline symbol, "escaped" by a \, and that record not being quoted. The file might look like this: Line1field1;Line1field2.1 \ Line1field2.2;Line1field3; Line2FIeld1;...
- 19140 Views
- 3 replies
- 1 kudos
- 13943 Views
- 7 replies
- 0 kudos
Resolved! How do I register a UDF that returns an array of tuples in scala/spark?
I'm relatively new to Scala. In the past, I was able to do the following python: def foo(p1, p2): import datetime as dt dt.datetime(2014, 4, 17, 12, 34) result = [ (1, "1", 1.1, dt.datetime(2014, 4, 17, 1, 0)), (2, "2", 2...
- 13943 Views
- 7 replies
- 0 kudos
- 0 kudos
Hello, Just in case, here is an example for proposed solution above: import org.apache.spark.sql.functions._ import org.apache.spark.sql.expressions._ import org.apache.spark.sql.types._ val data = Seq(("A", Seq((3,4),(5,6),(7,10))), ("B", Seq((-1,...
- 0 kudos
- 21975 Views
- 13 replies
- 1 kudos
How to add environment variable
Instead of setting the AWS accessKey and secret Key in hadoopConfiguration, I would like to add those in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. How can I do that in databricks?
- 21975 Views
- 13 replies
- 1 kudos
- 1 kudos
It is possible! I was able to confirm that the following post's "Best" answer works: https://forums.databricks.com/questions/11116/how-to-set-an-environment-variable.htmlFYI for @Miklos Christine​ and @Mike Trewartha​
- 1 kudos
- 43857 Views
- 2 replies
- 2 kudos
Pandas dataframe to a table
I want to write a pandas dataframe to a table, how can I do this ? Write command is not working, please help.
- 43857 Views
- 2 replies
- 2 kudos
- 2 kudos
Hey Kiran, Just taking a stab in the dark but do you want to convert the Pandas DataFrame to a Spark DataFrame and then write out the Spark DataFrame as a non-temporary SQL table? import pandas as pd ## Create Pandas Frame pd_df = pd.DataFrame({u'20...
- 2 kudos
- 6050 Views
- 4 replies
- 0 kudos
- 6050 Views
- 4 replies
- 0 kudos
- 0 kudos
df.na.replace(df.columns,Map("" -> "0")).show() // to convert from blank strings to zero
- 0 kudos
- 19864 Views
- 2 replies
- 1 kudos
How to increase spark.kryoserializer.buffer.max
when I join two dataframes, I got the following error. org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 1 Serialization trace: values (org.apache.spark.sql.catalyst.expressions.GenericRow) otherEle...
- 19864 Views
- 2 replies
- 1 kudos
- 1 kudos
val conf = new SparkConf() ... conf.set("spark.kryoserializer.buffer.max.mb", "512") ...
- 1 kudos
- 6823 Views
- 4 replies
- 0 kudos
- 6823 Views
- 4 replies
- 0 kudos
- 0 kudos
sample = (spark.read .format("com.databricks.spark.redshift") .option("url", jdbcUrl) .option("dbtable", "xx.xxx") # schema, table .option("forward_spark_s3_credentials", True) .option("tempdir", tem...
- 0 kudos
- 3252 Views
- 3 replies
- 0 kudos
Filtering records for all values of an array in Spark
I am very new to Spark. I have a very basic question. I have an array of values: listofECtokens: Array[String] = Array(EC-17A5206955089011B, EC-17A5206955089011A) I want to filter an RDD for all of these token values. I tried the following way: va...
- 3252 Views
- 3 replies
- 0 kudos
- 0 kudos
Actually, the intersection transformation does deduplication. If you don't need it, you can just slightly modify your code: val filteredRdd = rddAll.filter(line => line.contains(token)) and send data of the rdd to your program by calling of an act...
- 0 kudos
- 7784 Views
- 3 replies
- 0 kudos
calculate median and inter quartile range on spark dataframe
I have a spark dataframe of 5 columns and I want to calculate median and interquartile range on all. I am not able to figure out how do I write udf and call them on columns.
- 7784 Views
- 3 replies
- 0 kudos
- 0 kudos
Here is the easiest way to calculate this... https://stackoverflow.com/questions/37032689/scala-first-quartile-third-quartile-and-iqr-from-spark-sqlcontext-dataframe No Hive or windowing necessary.
- 0 kudos
- 6084 Views
- 1 replies
- 2 kudos
What is the difference between createTempView, createGlobalTempView and registerTempTable
Hi, friends! I have a question about difference between this three functions: dataframe . createTempViewdataframe . createGlobalTempView dataframe . registerTempTable all of them create intermediate tables. How to decide which I have to choose in c...
- 6084 Views
- 1 replies
- 2 kudos
- 2 kudos
From my understanding, createTempView (or more appropriately createOrReplaceTempView) has been introduced in Spark 2.0 to replace registerTempTable, which has been deprecated in 2.0. CreateTempView creates an in memory reference to the Dataframe in ...
- 2 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 456 | |
| 349 | |
| 287 |
