- 24358 Views
- 1 replies
- 0 kudos
How filter condition working in spark dataframe?
I have a table in hbase with 1 billions records.I want to filter the records based on certain condition (by date). For example: Dataframe.filter(col(date) === todayDate) Filter will be applied after all records from the table will be loaded into me...
- 24358 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @senthil kumar​ To pass external values to the filter (or where) transformations you can use the "lit" function in the following way:Dataframe.filter(col(date) == lit(todayDate))don´t know if that helps. Be careful with the schema infered by th...
- 0 kudos
- 18205 Views
- 4 replies
- 0 kudos
Issues with UTF-16 files and unicode characters
Can someone please offer some insight - I've spent days trying to solve this issue We have the task of loading in hundreds of tab seperated text files encoded in UTF-16 little endian with a tab delimiter. Our organisation is an international one and...
- 18205 Views
- 4 replies
- 0 kudos
- 0 kudos
You can also always read in the file as a textFile, and then run a UTF-16 decoder/encoder library as a UDF on the text.
- 0 kudos
- 15815 Views
- 8 replies
- 2 kudos
Resolved! Can I connect to a MS SQL server table in Databricks account?
I'd like to access a table on a MS SQL Server (Microsoft). Is it possible from Databricks? To my understanding, the syntax is something like this (in a SQL Notebook): CREATE TEMPORARY TABLE jdbcTable USING org.apache.spark.sql.jdbc OPTIONS ( url...
- 15815 Views
- 8 replies
- 2 kudos
- 2 kudos
Thanks for the trick that you have shared with us. I am really amazed to use this informational post. If you are facing MacBook error like MacBook Pro won't turn on black screen then click the link.
- 2 kudos
- 15824 Views
- 2 replies
- 0 kudos
Write data Frame into Azure Data Lake Storage
It happens that I am manipulating some data using Azure Databricks. Such data is in an Azure Data Lake Storage Gen1. I mounted the data into DBFS, but now, after transforming the data I would like to write it back into my data lake. To mount the dat...
- 15824 Views
- 2 replies
- 0 kudos
- 0 kudos
I am new in Azure Data Bricks..and I am trying to write the Data frame in mounted ADLS file. But in below command dfGPS.write.mode("overwrite").format("com.databricks.spark.csv").option("header","true").csv("/mnt/<mount-name>")
- 0 kudos
- 13329 Views
- 4 replies
- 0 kudos
Resolved! Error: TransportResponseHandler: Still have 1 requests outstanding when connection, occurring only on large dataset.
I am getting below error only during large dataset(i.e 15 TB compressed) . if my dataset is small( 1TB) i am not getting this error. Look like it fails on shuffle stage. Approx number of mappers is 150,000 Spark config:spark.sql.warehouse.dir hdfs:...
- 13329 Views
- 4 replies
- 0 kudos
- 20576 Views
- 4 replies
- 0 kudos
Writing large parquet file (500 millions row / 1000 columns) to S3 takes too much time
Hello community,First let me introduce my use case, i daily receive a 500 million rows like so :ID | Categories 1 | cat1, cat2, cat3, ..., catn 2 | cat1, catx, caty, ..., anothercategory Input data: 50 compressed csv files each file is 250 MB ...
- 20576 Views
- 4 replies
- 0 kudos
- 9937 Views
- 2 replies
- 0 kudos
why spark very slow with large number of dataframe columns
scala Spark App: I have a dataset of 130x14000. I read from a parquet file with SparkSession. Then used for Spark ML Random Forest model (using pipeline). It takes 7 hours to complete! for reading the parquet file takes about 1 minute. If I implemen...
- 9937 Views
- 2 replies
- 0 kudos
- 0 kudos
I've already answered a similar question on StackOverflow so I'll repeat what a I said there. The following may not solve your problem completely but it should give you some pointer to start. The first problem that you are facing is the disproportio...
- 0 kudos
- 8463 Views
- 1 replies
- 0 kudos
How to store a pyspark dataframe in S3 bucket.
I have a pyspark dataframe df containing 4 columns. How can I write this dataframe to s3 bucket? I'm using pycharm to execute the code. and what are the packages required to be installed?
- 8463 Views
- 1 replies
- 0 kudos
- 0 kudos
You shouldn't need any packages. You can mount S3 bucket to Databricks cluster. https://docs.databricks.com/spark/latest/data-sources/aws/amazon-s3.html#mount-aws-s3 or this http://www.sparktutorials.net/Reading+and+Writing+S3+Data+with+Apache+Spark...
- 0 kudos
- 33521 Views
- 8 replies
- 1 kudos
Resolved! Comparing two dataframes
How can we compare two data frames using pyspark I need to validate my output with another dataset
- 33521 Views
- 8 replies
- 1 kudos
- 1 kudos
I think the best bet in such a case is to take inner join (equivalent to intersection) by putting a condition on those columns which necessarily need to have same value in both dataframes. For example, let df1 and df2 are two dataframes. df1 has co...
- 1 kudos
- 15897 Views
- 5 replies
- 0 kudos
How to prevent spark-csv from adding quotes to JSON string in dataframe
I have a sql dataframe with a column that has a json string in it (e.g. {"key":"value"}). When I use spark-csv to save the dataframe it changes the field values to be "{""key"":""valule""}". Is there a way to turn that off?
- 15897 Views
- 5 replies
- 0 kudos
- 0 kudos
Do quote or escape options only work with "Write" instead of "read"? Our source files contain doube quotes. We'd like to add backsplash (escape) in front each double quote before converting the values from out dataframes to json outputs.
- 0 kudos
- 6853 Views
- 1 replies
- 0 kudos
How to move files of same extension in databricks files system?
I am facing file not found exception when i am trying to move the file with * in DBFS. Here both source and destination directories are in DBFS. I have the source file named "test_sample.csv" available in dbfs directory and i am using the command li...
- 6853 Views
- 1 replies
- 0 kudos
- 0 kudos
@bkr, you can reference the file name using dbutils and then pass this to the move command. Here's an example for this in Scala: val fileNm = dbutils.fs.ls("/usr/krishna/sample").map(_.name).filter(r => r.startsWith("test"))(0) val fileLoc = "dbfs:/...
- 0 kudos
- 9792 Views
- 5 replies
- 0 kudos
Resolved! How do I get a cartesian product of a huge dataset?
A cartesian product is a common operation to get the cross product of two tables. For example, say you have a list of customers and a list of your product catalog and want to get the cross product of all customer - product combinations. Cartesian pr...
- 9792 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi buddies, it is great written piece entirely defined, continue the good work constantly.
- 0 kudos
- 6161 Views
- 3 replies
- 0 kudos
Resolving NoClassDefFoundError: org/apache/spark/Logging exception
I was trying out hbase-spark connector. To start with, I am trying out this code. My pom dependencies are: <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version...
- 6161 Views
- 3 replies
- 0 kudos
- 0 kudos
The alpha of hbase-spark you're using depends on Spark 1.6 -- see hbase-spark/pom.xml:L33 -- so you'll probably have to stick with 1.6 if you want to use that published jar. For reasons I don't understand hbase-spark was removed in the last couple o...
- 0 kudos
- 17702 Views
- 4 replies
- 0 kudos
How to convert column type from str to date in sparksql when the format is not yyyy-mm-dd?
I imported a large csv file into databricks as a table. I am able to run sql queries on it in a databricks notebook. In my table, I have a column that contains date information in the mm/dd/yyyy format : 12/29/2015 12/30/2015 etc... Databricks impo...
- 17702 Views
- 4 replies
- 0 kudos
- 0 kudos
@josephpconley would it be safe to cast a column that contains null values?
- 0 kudos
- 19961 Views
- 2 replies
- 0 kudos
upload local files into DBFS
I am using Databricks Notebook Community Edition (2.36) and want to upload a local file into DBFS. Is there any simple Hadoop commands like "hadoop fs -put ..."? Any help would be appreciated.
- 19961 Views
- 2 replies
- 0 kudos
- 0 kudos
Python 2.7:import urllib.request urllib.urlretrieve("https://github.com/sushrutt12/DataSets/blob/master/final_chris.zip","/tmp/chris_data.zip") dbutils.fs.mv("file:/tmp/chris_data.zip", "dbfs:/data/chris_data.zip")Python 3.x: import urllib.requesturl...
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 456 | |
| 349 | |
| 287 |
