- 12510 Views
- 2 replies
- 0 kudos
Resolved! How to Change Schema of a Spark SQL
I am new to Spark and just started an online pyspark tutorial. I uploaded the json data in DataBrick and wrote the commands as follows: df = sqlContext.sql("SELECT * FROM people_json") df.printSchema() from pyspark.sql.types import * data_schema =...
- 12510 Views
- 2 replies
- 0 kudos
- 10947 Views
- 5 replies
- 0 kudos
Why Databricks spark is faster than AWS EMR Spark ?
https://databricks.com/blog/2017/07/12/benchmarking-big-data-sql-platforms-in-the-cloud.html Hi All, just wondering why Databricks Spark is lot faster on S3 compared with AWS EMR spark both the systems are on spark version 2.4 , is Databricks have ...
- 10947 Views
- 5 replies
- 0 kudos
- 0 kudos
I think you can get some pretty good insight into the optimizations on Databricks here:https://docs.databricks.com/delta/delta-on-databricks.html Specifically, check out the sections on caching, z-ordering, and join optimization. There's also a grea...
- 0 kudos
- 9427 Views
- 1 replies
- 0 kudos
How to create a surrogate key sequence which I can use in SCD cases?
Hi Community I would like to know if there is an option to create an integer sequence which persists even if the cluster is shut down. My target is to use this integer value as a surrogate key to join different tables or do Slowly changing dimensio...
- 9427 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pascalvanbellen ,There is no concept of FK, PK, SK in Spark. But Databricks Delta automatically takes care of SCD type scenarios. https://docs.databricks.com/spark/latest/spark-sql/language-manual/merge-into.html#slowly-changing-data-scd-type-2 ...
- 0 kudos
- 8654 Views
- 4 replies
- 0 kudos
Splitting Date into Year, Month and Day, with inconsistent delimiters
I am trying to split my Date Column which is a String Type right now into 3 columns Year, Month and Date. I use (PySpark): <code>split_date=pyspark.sql.functions.split(df['Date'], '-') df= df.withColumn('Year', split_date.getItem(0)) df= df.wit...
- 8654 Views
- 4 replies
- 0 kudos
- 24320 Views
- 1 replies
- 0 kudos
How filter condition working in spark dataframe?
I have a table in hbase with 1 billions records.I want to filter the records based on certain condition (by date). For example: Dataframe.filter(col(date) === todayDate) Filter will be applied after all records from the table will be loaded into me...
- 24320 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @senthil kumar​ To pass external values to the filter (or where) transformations you can use the "lit" function in the following way:Dataframe.filter(col(date) == lit(todayDate))don´t know if that helps. Be careful with the schema infered by th...
- 0 kudos
- 15765 Views
- 8 replies
- 2 kudos
Resolved! Can I connect to a MS SQL server table in Databricks account?
I'd like to access a table on a MS SQL Server (Microsoft). Is it possible from Databricks? To my understanding, the syntax is something like this (in a SQL Notebook): CREATE TEMPORARY TABLE jdbcTable USING org.apache.spark.sql.jdbc OPTIONS ( url...
- 15765 Views
- 8 replies
- 2 kudos
- 2 kudos
Thanks for the trick that you have shared with us. I am really amazed to use this informational post. If you are facing MacBook error like MacBook Pro won't turn on black screen then click the link.
- 2 kudos
- 13301 Views
- 4 replies
- 0 kudos
Resolved! Error: TransportResponseHandler: Still have 1 requests outstanding when connection, occurring only on large dataset.
I am getting below error only during large dataset(i.e 15 TB compressed) . if my dataset is small( 1TB) i am not getting this error. Look like it fails on shuffle stage. Approx number of mappers is 150,000 Spark config:spark.sql.warehouse.dir hdfs:...
- 13301 Views
- 4 replies
- 0 kudos
- 20543 Views
- 4 replies
- 0 kudos
Writing large parquet file (500 millions row / 1000 columns) to S3 takes too much time
Hello community,First let me introduce my use case, i daily receive a 500 million rows like so :ID | Categories 1 | cat1, cat2, cat3, ..., catn 2 | cat1, catx, caty, ..., anothercategory Input data: 50 compressed csv files each file is 250 MB ...
- 20543 Views
- 4 replies
- 0 kudos
- 9922 Views
- 2 replies
- 0 kudos
why spark very slow with large number of dataframe columns
scala Spark App: I have a dataset of 130x14000. I read from a parquet file with SparkSession. Then used for Spark ML Random Forest model (using pipeline). It takes 7 hours to complete! for reading the parquet file takes about 1 minute. If I implemen...
- 9922 Views
- 2 replies
- 0 kudos
- 0 kudos
I've already answered a similar question on StackOverflow so I'll repeat what a I said there. The following may not solve your problem completely but it should give you some pointer to start. The first problem that you are facing is the disproportio...
- 0 kudos
- 6144 Views
- 3 replies
- 0 kudos
Resolving NoClassDefFoundError: org/apache/spark/Logging exception
I was trying out hbase-spark connector. To start with, I am trying out this code. My pom dependencies are: <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version...
- 6144 Views
- 3 replies
- 0 kudos
- 0 kudos
The alpha of hbase-spark you're using depends on Spark 1.6 -- see hbase-spark/pom.xml:L33 -- so you'll probably have to stick with 1.6 if you want to use that published jar. For reasons I don't understand hbase-spark was removed in the last couple o...
- 0 kudos
- 11277 Views
- 0 replies
- 0 kudos
TypeError: Column is not iterable when using more than one columns in withColumn()
I am trying to find quarter start date from a date column. I get the expected result when i write it using selectExpr() but when i add the same logic in .withColumn() i get TypeError: Column is not iterableI am using a workaround as follows workarou...
- 11277 Views
- 0 replies
- 0 kudos
- 6427 Views
- 1 replies
- 0 kudos
I want to split a dataframe with date range 1 week, with each week data in different column.
DF Q Date(yyyy-mm-dd) q1 2017-10-01 q2 2017-10-03 q1 2017-10-09 q3 2017-10-06 q2 2017-10-01 q1 2017-10-13 Q1 2017-10-02 Q3 2017-10-21 Q4 2017-10-17 Q5 2017-10-20 Q4 2017-10-31 Q2 2017-10-27 Q5 2017-10-01 Dataframe: ...
- 6427 Views
- 1 replies
- 0 kudos
- 0 kudos
It should just be a matter of applying the correct set of transformations:You can start by adding the week-of-year to each record with the command pyspark.sql.functions.weekofyear(..) and name it something like weekOfYear. See https://spark.apache.or...
- 0 kudos
- 19096 Views
- 3 replies
- 1 kudos
Resolved! How to parse a file with newline character, escaped with \ and not quoted
Hi! I am facing an issue when reading and parsing a CSV file. Some records have a newline symbol, "escaped" by a \, and that record not being quoted. The file might look like this: Line1field1;Line1field2.1 \ Line1field2.2;Line1field3; Line2FIeld1;...
- 19096 Views
- 3 replies
- 1 kudos
- 13899 Views
- 7 replies
- 0 kudos
Resolved! How do I register a UDF that returns an array of tuples in scala/spark?
I'm relatively new to Scala. In the past, I was able to do the following python: def foo(p1, p2): import datetime as dt dt.datetime(2014, 4, 17, 12, 34) result = [ (1, "1", 1.1, dt.datetime(2014, 4, 17, 1, 0)), (2, "2", 2...
- 13899 Views
- 7 replies
- 0 kudos
- 0 kudos
Hello, Just in case, here is an example for proposed solution above: import org.apache.spark.sql.functions._ import org.apache.spark.sql.expressions._ import org.apache.spark.sql.types._ val data = Seq(("A", Seq((3,4),(5,6),(7,10))), ("B", Seq((-1,...
- 0 kudos
- 19850 Views
- 2 replies
- 1 kudos
How to increase spark.kryoserializer.buffer.max
when I join two dataframes, I got the following error. org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 1 Serialization trace: values (org.apache.spark.sql.catalyst.expressions.GenericRow) otherEle...
- 19850 Views
- 2 replies
- 1 kudos
- 1 kudos
val conf = new SparkConf() ... conf.set("spark.kryoserializer.buffer.max.mb", "512") ...
- 1 kudos