- 29716 Views
- 11 replies
- 0 kudos
Dataframe Write Append to Parquet Table - Partition Issue
Hello, I am attempting to append new json files into an existing parquet table defined in Databricks. Using a dataset defined by this command (dataframe initially added to a temp table): val output = sql("select headers.event_name, to_date(from_unix...
- 29716 Views
- 11 replies
- 0 kudos
- 0 kudos
We came across similar situation we are using spark 1.6.1, we have a daily load process to pull data from oracle and write as parquet files, this works fine for 18 days of data (till 18th run), the problem comes after 19th run where the data frame l...
- 0 kudos
- 10234 Views
- 3 replies
- 0 kudos
Resolved! How to see the textbox input from getArgument() or dbutils.widgets.text() or dbutils.widgets.dropdown()
getArgument() has been deprecated. I don't see the text box for me to type in any input anymore. What I actually see though is the following error: Deprecation warning: Use dbutils.widgets.text() or dbutils.widgets.dropdown() to create a widget and...
- 10234 Views
- 3 replies
- 0 kudos
- 0 kudos
So shouldn't it be removed from the tutorial notebook showing how to connect to S3? I'm trying to connect to S3 for the first time and a deprecation warning isn't a pleasant first experience with a tool I am paying for.
- 0 kudos
- 17100 Views
- 5 replies
- 0 kudos
Create a in-memory table in Spark and insert data into it
Hi, My requirement is I need to create a Spark In-memory table (Not pushing hive table into memory) insert data into it and finally write that back to Hive table. Idea here is to avoid the disk IO while writing into Target Hive table. There are lot ...
- 17100 Views
- 5 replies
- 0 kudos
- 0 kudos
Got it - how about using a UnionAll? I believe this code snippet does what you'd want:from pyspark.sql import Row array = [Row(value=1), Row(value=2), Row(value=3)] df = sqlContext.createDataFrame(sc.parallelize(array)) array2 = [Row(value=4), Ro...
- 0 kudos
- 5177 Views
- 1 replies
- 1 kudos
sql: how to convert datatype of column?
Bricklayers, I want to port this sql statement from sqlite to databricks: select cast(myage as number) as my_integer_age from ages; Does databricks allow me to do something like this?
- 5177 Views
- 1 replies
- 1 kudos
- 1 kudos
@dan11 We don't support number in Spark SQL. Try using int, double, float, and your query should be fine. To run SQL in a notebook, just prepend any cell with %sql. %sql select cast(myage as double) as my_integer_age from ages;
- 1 kudos
- 12594 Views
- 1 replies
- 0 kudos
- 12594 Views
- 1 replies
- 0 kudos
- 0 kudos
got it use - features = dataset.map(lambda row: row[0:]) from pyspark.mllib.stat import Statistics corr_mat=Statistics.corr(features, method="pearson")
- 0 kudos
- 17975 Views
- 5 replies
- 0 kudos
Spark + Python - Java gateway process exited before sending the driver its port number?
Why do I get this error on my browser screen, <type 'exceptions.Exception'>: Java gateway process exited before sending the driver its port number args = ('Java gateway process exited before sending the driver its port number',) message = 'Java gat...
- 17975 Views
- 5 replies
- 0 kudos
- 0 kudos
I'm facing the same problem, does anybody know how to connect Spark in Ipython notebook? The issue I created, https://github.com/jupyter/notebook/issues/743
- 0 kudos
- 15379 Views
- 2 replies
- 1 kudos
How can I use display() in a python notebook with pyspark.sql.Row Objects, e.g. after calling the first() operation on a DataFrame?
I'm trying to display() the results from calling first() on a DataFrame, but display() doesn't work with pyspark.sql.Row objects. How can I display this result?
- 15379 Views
- 2 replies
- 1 kudos
- 20194 Views
- 8 replies
- 0 kudos
My Spark SQL join is very slow - what can I do to speed it up?
It's taking 10-12 minutes - can I make it faster?
- 20194 Views
- 8 replies
- 0 kudos
- 0 kudos
Analyze is not needed with parquet tables that use the databricks parquet package. That is the default now when you use .saveAsTable(), but if you use a different output format - it's possible that analyze may not work yet.
- 0 kudos
- 7344 Views
- 1 replies
- 0 kudos
java.lang.OutOfMemoryError: GC overhead limit exceeded
I get java.lang.OutOfMemoryError: GC overhead limit exceeded when trying coutn action on a file. The file is a CSV file 217GB zise Im using a 10 r3.8xlarge(ubuntu) machines cdh 5.3.6 and spark 1.2.0 configutation: spark.app.id:local-1443956477103 s...
- 7344 Views
- 1 replies
- 0 kudos
- 0 kudos
Looks like the following property is pretty high, which consumes a lot of memory on your executors when you cache the dataset. "spark.storage.memoryFraction:0.9" This could likely be solved by changing the configuration. Take a look at the upstream...
- 0 kudos
- 11675 Views
- 1 replies
- 0 kudos
- 11675 Views
- 1 replies
- 0 kudos
- 0 kudos
There is no hardcoded limit we just call panda.fromRecords with a collection of fields to instantiate a new Panda Dataframe. The only limit is memory. See http://stackoverflow.com/questions/15455722/pandas-is-there-a-max-size-max-no-of-columns-max-r...
- 0 kudos
- 14409 Views
- 3 replies
- 0 kudos
- 14409 Views
- 3 replies
- 0 kudos
- 12160 Views
- 1 replies
- 1 kudos
- 12160 Views
- 1 replies
- 1 kudos
- 1 kudos
Hover between the cells in the side-to-side middle and you will see a + sign appear. This is how you can insert cells into the top-to-bottom middle of a notebook. You can also move cells by hovering in the upper left of each cell. A cross-hairs will...
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 573 | |
| 349 | |
| 287 |
