cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RobertWalsh
by New Contributor II
  • 13164 Views
  • 11 replies
  • 0 kudos

Dataframe Write Append to Parquet Table - Partition Issue

Hello, I am attempting to append new json files into an existing parquet table defined in Databricks. Using a dataset defined by this command (dataframe initially added to a temp table): val output = sql("select headers.event_name, to_date(from_unix...

0693f000007OoJYAA0 0693f000007OoJZAA0
  • 13164 Views
  • 11 replies
  • 0 kudos
Latest Reply
anil_s_langote
New Contributor II
  • 0 kudos

We came across similar situation we are using spark 1.6.1, we have a daily load process to pull data from oracle and write as parquet files, this works fine for 18 days of data (till 18th run), the problem comes after 19th run where the data frame l...

  • 0 kudos
10 More Replies
jpalbeza
by New Contributor II
  • 4580 Views
  • 3 replies
  • 0 kudos

Resolved! How to see the textbox input from getArgument() or dbutils.widgets.text() or dbutils.widgets.dropdown()

getArgument() has been deprecated. I don't see the text box for me to type in any input anymore. What I actually see though is the following error: Deprecation warning: Use dbutils.widgets.text() or dbutils.widgets.dropdown() to create a widget and...

  • 4580 Views
  • 3 replies
  • 0 kudos
Latest Reply
RyanJohnson
New Contributor II
  • 0 kudos

So shouldn't it be removed from the tutorial notebook showing how to connect to S3? I'm trying to connect to S3 for the first time and a deprecation warning isn't a pleasant first experience with a tool I am paying for.

  • 0 kudos
2 More Replies
Sri1
by New Contributor II
  • 7156 Views
  • 5 replies
  • 0 kudos

Create a in-memory table in Spark and insert data into it

Hi, My requirement is I need to create a Spark In-memory table (Not pushing hive table into memory) insert data into it and finally write that back to Hive table. Idea here is to avoid the disk IO while writing into Target Hive table. There are lot ...

  • 7156 Views
  • 5 replies
  • 0 kudos
Latest Reply
vida
Contributor II
  • 0 kudos

Got it - how about using a UnionAll? I believe this code snippet does what you'd want:from pyspark.sql import Row array = [Row(value=1), Row(value=2), Row(value=3)] df = sqlContext.createDataFrame(sc.parallelize(array)) array2 = [Row(value=4), Ro...

  • 0 kudos
4 More Replies
dan11
by New Contributor II
  • 3000 Views
  • 1 replies
  • 1 kudos

sql: how to convert datatype of column?

Bricklayers, I want to port this sql statement from sqlite to databricks: select cast(myage as number) as my_integer_age from ages; Does databricks allow me to do something like this?

  • 3000 Views
  • 1 replies
  • 1 kudos
Latest Reply
raela
New Contributor III
  • 1 kudos

@dan11 We don't support number in Spark SQL. Try using int, double, float, and your query should be fine. To run SQL in a notebook, just prepend any cell with %sql. %sql select cast(myage as double) as my_integer_age from ages;

  • 1 kudos
washim
by New Contributor III
  • 8305 Views
  • 1 replies
  • 0 kudos
  • 8305 Views
  • 1 replies
  • 0 kudos
Latest Reply
washim
New Contributor III
  • 0 kudos

got it use - features = dataset.map(lambda row: row[0:]) from pyspark.mllib.stat import Statistics corr_mat=Statistics.corr(features, method="pearson")

  • 0 kudos
lau_thiamkok
by New Contributor II
  • 10793 Views
  • 5 replies
  • 0 kudos

Spark + Python - Java gateway process exited before sending the driver its port number?

Why do I get this error on my browser screen, <type 'exceptions.Exception'>: Java gateway process exited before sending the driver its port number args = ('Java gateway process exited before sending the driver its port number',) message = 'Java gat...

  • 10793 Views
  • 5 replies
  • 0 kudos
Latest Reply
EricaLi
New Contributor II
  • 0 kudos

I'm facing the same problem, does anybody know how to connect Spark in Ipython notebook? The issue I created, https://github.com/jupyter/notebook/issues/743

  • 0 kudos
4 More Replies
Anonymous
by Not applicable
  • 9199 Views
  • 2 replies
  • 0 kudos

How can I use display() in a python notebook with pyspark.sql.Row Objects, e.g. after calling the first() operation on a DataFrame?

I'm trying to display() the results from calling first() on a DataFrame, but display() doesn't work with pyspark.sql.Row objects. How can I display this result?

  • 9199 Views
  • 2 replies
  • 0 kudos
Latest Reply
dnchari
New Contributor II
  • 0 kudos

Use take()

  • 0 kudos
1 More Replies
vida
by Contributor II
  • 6990 Views
  • 8 replies
  • 0 kudos

My Spark SQL join is very slow - what can I do to speed it up?

It's taking 10-12 minutes - can I make it faster?

  • 6990 Views
  • 8 replies
  • 0 kudos
Latest Reply
vida
Contributor II
  • 0 kudos

Analyze is not needed with parquet tables that use the databricks parquet package. That is the default now when you use .saveAsTable(), but if you use a different output format - it's possible that analyze may not work yet.

  • 0 kudos
7 More Replies
t_ras
by New Contributor
  • 4923 Views
  • 1 replies
  • 0 kudos

java.lang.OutOfMemoryError: GC overhead limit exceeded

I get java.lang.OutOfMemoryError: GC overhead limit exceeded when trying coutn action on a file. The file is a CSV file 217GB zise Im using a 10 r3.8xlarge(ubuntu) machines cdh 5.3.6 and spark 1.2.0 configutation: spark.app.id:local-1443956477103 s...

  • 4923 Views
  • 1 replies
  • 0 kudos
Latest Reply
miklos
Contributor
  • 0 kudos

Looks like the following property is pretty high, which consumes a lot of memory on your executors when you cache the dataset. "spark.storage.memoryFraction:0.9" This could likely be solved by changing the configuration. Take a look at the upstream...

  • 0 kudos
Gabriela_DeQuer
by New Contributor
  • 5863 Views
  • 1 replies
  • 0 kudos
  • 5863 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826991422
Contributor
  • 0 kudos

There is no hardcoded limit we just call panda.fromRecords with a collection of fields to instantiate a new Panda Dataframe. The only limit is memory. See http://stackoverflow.com/questions/15455722/pandas-is-there-a-max-size-max-no-of-columns-max-r...

  • 0 kudos
cfregly
by Contributor
  • 8620 Views
  • 1 replies
  • 0 kudos
  • 8620 Views
  • 1 replies
  • 0 kudos
Latest Reply
cfregly
Contributor
  • 0 kudos

Sorted DataIf your data is sorted using either sort() or ORDER BY, these operations will be deterministic and return either the 1st element using first()/head() or the top-n using head(n)/take(n).show()/show(n) return Unit (void) and will print up to...

  • 0 kudos
__Databricks_Su
by Contributor
  • 7682 Views
  • 1 replies
  • 1 kudos
  • 7682 Views
  • 1 replies
  • 1 kudos
Latest Reply
__Databricks_Su
Contributor
  • 1 kudos

Hover between the cells in the side-to-side middle and you will see a + sign appear. This is how you can insert cells into the top-to-bottom middle of a notebook. You can also move cells by hovering in the upper left of each cell. A cross-hairs will...

  • 1 kudos