- 33547 Views
- 8 replies
- 1 kudos
Resolved! Comparing two dataframes
How can we compare two data frames using pyspark I need to validate my output with another dataset
- 33547 Views
- 8 replies
- 1 kudos
- 1 kudos
I think the best bet in such a case is to take inner join (equivalent to intersection) by putting a condition on those columns which necessarily need to have same value in both dataframes. For example, let df1 and df2 are two dataframes. df1 has co...
- 1 kudos
- 15913 Views
- 5 replies
- 0 kudos
How to prevent spark-csv from adding quotes to JSON string in dataframe
I have a sql dataframe with a column that has a json string in it (e.g. {"key":"value"}). When I use spark-csv to save the dataframe it changes the field values to be "{""key"":""valule""}". Is there a way to turn that off?
- 15913 Views
- 5 replies
- 0 kudos
- 0 kudos
Do quote or escape options only work with "Write" instead of "read"? Our source files contain doube quotes. We'd like to add backsplash (escape) in front each double quote before converting the values from out dataframes to json outputs.
- 0 kudos
- 6858 Views
- 1 replies
- 0 kudos
How to move files of same extension in databricks files system?
I am facing file not found exception when i am trying to move the file with * in DBFS. Here both source and destination directories are in DBFS. I have the source file named "test_sample.csv" available in dbfs directory and i am using the command li...
- 6858 Views
- 1 replies
- 0 kudos
- 0 kudos
@bkr, you can reference the file name using dbutils and then pass this to the move command. Here's an example for this in Scala: val fileNm = dbutils.fs.ls("/usr/krishna/sample").map(_.name).filter(r => r.startsWith("test"))(0) val fileLoc = "dbfs:/...
- 0 kudos
- 9821 Views
- 5 replies
- 0 kudos
Resolved! How do I get a cartesian product of a huge dataset?
A cartesian product is a common operation to get the cross product of two tables. For example, say you have a list of customers and a list of your product catalog and want to get the cross product of all customer - product combinations. Cartesian pr...
- 9821 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi buddies, it is great written piece entirely defined, continue the good work constantly.
- 0 kudos
- 6175 Views
- 3 replies
- 0 kudos
Resolving NoClassDefFoundError: org/apache/spark/Logging exception
I was trying out hbase-spark connector. To start with, I am trying out this code. My pom dependencies are: <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version...
- 6175 Views
- 3 replies
- 0 kudos
- 0 kudos
The alpha of hbase-spark you're using depends on Spark 1.6 -- see hbase-spark/pom.xml:L33 -- so you'll probably have to stick with 1.6 if you want to use that published jar. For reasons I don't understand hbase-spark was removed in the last couple o...
- 0 kudos
- 17715 Views
- 4 replies
- 0 kudos
How to convert column type from str to date in sparksql when the format is not yyyy-mm-dd?
I imported a large csv file into databricks as a table. I am able to run sql queries on it in a databricks notebook. In my table, I have a column that contains date information in the mm/dd/yyyy format : 12/29/2015 12/30/2015 etc... Databricks impo...
- 17715 Views
- 4 replies
- 0 kudos
- 0 kudos
@josephpconley would it be safe to cast a column that contains null values?
- 0 kudos
- 19965 Views
- 2 replies
- 0 kudos
upload local files into DBFS
I am using Databricks Notebook Community Edition (2.36) and want to upload a local file into DBFS. Is there any simple Hadoop commands like "hadoop fs -put ..."? Any help would be appreciated.
- 19965 Views
- 2 replies
- 0 kudos
- 0 kudos
Python 2.7:import urllib.request urllib.urlretrieve("https://github.com/sushrutt12/DataSets/blob/master/final_chris.zip","/tmp/chris_data.zip") dbutils.fs.mv("file:/tmp/chris_data.zip", "dbfs:/data/chris_data.zip")Python 3.x: import urllib.requesturl...
- 0 kudos
- 12476 Views
- 12 replies
- 0 kudos
Resolved! Custom line separator
I see that https://github.com/apache/spark/pull/18581 will enable defining custom Line Separators for many sources, including CSV. Apart from waiting on this PR to make it into the main Databricks runtime, is there any other alternative to support d...
- 12476 Views
- 12 replies
- 0 kudos
- 0 kudos
You can use newAPIHadoopFile SCALA import org.apache.hadoop.io.LongWritable import org.apache.hadoop.io.Text import org.apache.hadoop.conf.Configuration import org.apache.hadoop.mapreduce.lib.input.TextInputFormat val conf = new Configuration conf.s...
- 0 kudos
- 18412 Views
- 3 replies
- 0 kudos
Resolved! I've set the partition mode to nonstrict in hive but spark is not seeing it
I've got a table I want to add some data to and it's partitoned. I want to use dynamic partitioning but I get this error org.apache.spark.SparkException: Dynamic partition strict mode requires at least one static partition column. To turn this off ...
- 18412 Views
- 3 replies
- 0 kudos
- 0 kudos
I got it working. This was exactly what I needed. Thank you @Peyman Mohajerian​
- 0 kudos
- 11296 Views
- 0 replies
- 0 kudos
TypeError: Column is not iterable when using more than one columns in withColumn()
I am trying to find quarter start date from a date column. I get the expected result when i write it using selectExpr() but when i add the same logic in .withColumn() i get TypeError: Column is not iterableI am using a workaround as follows workarou...
- 11296 Views
- 0 replies
- 0 kudos
- 6459 Views
- 1 replies
- 0 kudos
I want to split a dataframe with date range 1 week, with each week data in different column.
DF Q Date(yyyy-mm-dd) q1 2017-10-01 q2 2017-10-03 q1 2017-10-09 q3 2017-10-06 q2 2017-10-01 q1 2017-10-13 Q1 2017-10-02 Q3 2017-10-21 Q4 2017-10-17 Q5 2017-10-20 Q4 2017-10-31 Q2 2017-10-27 Q5 2017-10-01 Dataframe: ...
- 6459 Views
- 1 replies
- 0 kudos
- 0 kudos
It should just be a matter of applying the correct set of transformations:You can start by adding the week-of-year to each record with the command pyspark.sql.functions.weekofyear(..) and name it something like weekOfYear. See https://spark.apache.or...
- 0 kudos
- 36930 Views
- 0 replies
- 2 kudos
Requesting support for "SELECT TOP n from Table"
In notebook, It looks like if I need to select top N rows, I can rely on "LIMIT" keyword. It would be nice if you can support "TOP" as well The current approach to select 10 rows: select * from table1 LIMIT 10 Requesting TOP support: SELECT TOP 10 *...
- 36930 Views
- 0 replies
- 2 kudos
- 19177 Views
- 3 replies
- 1 kudos
Resolved! How to parse a file with newline character, escaped with \ and not quoted
Hi! I am facing an issue when reading and parsing a CSV file. Some records have a newline symbol, "escaped" by a \, and that record not being quoted. The file might look like this: Line1field1;Line1field2.1 \ Line1field2.2;Line1field3; Line2FIeld1;...
- 19177 Views
- 3 replies
- 1 kudos
- 13970 Views
- 7 replies
- 0 kudos
Resolved! How do I register a UDF that returns an array of tuples in scala/spark?
I'm relatively new to Scala. In the past, I was able to do the following python: def foo(p1, p2): import datetime as dt dt.datetime(2014, 4, 17, 12, 34) result = [ (1, "1", 1.1, dt.datetime(2014, 4, 17, 1, 0)), (2, "2", 2...
- 13970 Views
- 7 replies
- 0 kudos
- 0 kudos
Hello, Just in case, here is an example for proposed solution above: import org.apache.spark.sql.functions._ import org.apache.spark.sql.expressions._ import org.apache.spark.sql.types._ val data = Seq(("A", Seq((3,4),(5,6),(7,10))), ("B", Seq((-1,...
- 0 kudos
- 22080 Views
- 13 replies
- 1 kudos
How to add environment variable
Instead of setting the AWS accessKey and secret Key in hadoopConfiguration, I would like to add those in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. How can I do that in databricks?
- 22080 Views
- 13 replies
- 1 kudos
- 1 kudos
It is possible! I was able to confirm that the following post's "Best" answer works: https://forums.databricks.com/questions/11116/how-to-set-an-environment-variable.htmlFYI for @Miklos Christine​ and @Mike Trewartha​
- 1 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 466 | |
| 349 | |
| 287 |
