Databricks

schnee1 · 10-23-2015

I have JSON data set that contains a price in a string like "USD 5.00". I'd like to convert the numeric portion to a Double to use in an MLLIB LabeledPoint, and have managed to split the price string into an array of string. The below creates a data...

schnee1 · 05-09-2016

I recently watched a webinar in which @rxin clear the results from the Javascript Console (in Chrome) View -> Developer -> JavaScript Console and then type "notebook.clearResults()" The webinar was about Spark 2.0, which was great, but that little b...

schnee1 · 10-24-2015

For a bit more detail, this sort of worked: val dfExploded = df.explode(df("price")) { case Row(pr: WrappedArray[String]) => pr.map(pr => Price(pr(0).toString, pr(1).toString) ) } dfExploded.show() (I had to use "WrappedArray" instead of "Array" to...

schnee1 · 10-24-2015

I wound up getting past it with something like: val assembler = new VectorAssembler() .setInputCols(Array("f1", "f2")) .setOutputCol("features") val labeled = assembler.transform(df) .select($"price".getItem(1).cast("double"), $"features") .map{cas...

schnee1 · 10-23-2015

thanks @cfregly. I think I must need a lot more remedial learning on Scala. The reference you provided works great, but when I try to translate it into my problem via: val dfExploded = df.explode(df("price")) { case Row(pr: Array[Row]) => pr.map(pr ...

Databricks

User Stats

User Activity

Access struct elements inside dataframe?

Re: How do I clear all output results in a notebook?

Re: Access struct elements inside dataframe?

Re: Access struct elements inside dataframe?

Re: Access struct elements inside dataframe?