- 9188 Views
- 5 replies
- 4 kudos
Read external iceberg table in a spark dataframe within databricks
I am trying to read an external iceberg database from s3 location using the follwing commanddf_source = (spark.read.format("iceberg") .load(source_s3_path) .drop(*source_drop_columns) .filter(f"{date_column}<='{date_filter}'") )B...
- 9188 Views
- 5 replies
- 4 kudos
- 5552 Views
- 1 replies
- 0 kudos
While trying to save the spark dataframe to delta table is taking too long
While working on video analytics task I need to save the image bytes to the delta table earlier extracted into the spark dataframe. While I want to over write a same delta table over the period of complete task and also the size of input data differs...
- 5552 Views
- 1 replies
- 0 kudos
- 0 kudos
can you check the spark UI, to see where the time is spent?It can be a join, udf, ...
- 0 kudos
- 813 Views
- 0 replies
- 0 kudos
Spark DataFrame apply Databricks geospatial indexing functions
I have a spark DataFrame with `h3` hex ids and I am trying to obtain the polygon geometries. from pyspark.sql import SparkSession from pyspark.sql.functions import col, expr from pyspark.databricks.sql.functions import * from mosaic import enable_m...
- 813 Views
- 0 replies
- 0 kudos
- 1836 Views
- 2 replies
- 2 kudos
To collect the elements of a SparkDataFrame and coerces them into an R dataframe.
Hello Everyone,I am facing the challenge while collecting a spark dataframe into an R dataframe, this I need to do as I am using TraMineR algorithm whih is implemented in R only and the data pre-processing I have done in pysparkI am trying this:event...
- 1836 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Niraj Tanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 2 kudos
- 3174 Views
- 1 replies
- 1 kudos
Save Spark DataFrame to shape file (.shp format)
Hello,I know how to create .shp file from Geopandas dataframe using code similar to this, also mentioned on SO:gpd_df = geopandas.GeoDataFrame(pandas_df, geometry='geom') gpd_df .to_file("username/nh.shp")However I have .parquet files that I can load...
- 3174 Views
- 1 replies
- 1 kudos
- 1 kudos
@Bartosz Maciejewski​ :Spark does not have native support for writing Shapefiles directly. However, you can use a third-party library such as GeoPandas or PyShp to write your Spark DataFrame to a Shapefile.Here's an example of how to use GeoPandas to...
- 1 kudos
- 6833 Views
- 4 replies
- 3 kudos
Resolved! Writing PySpark DataFrame onto AWS Glue throwing error
I have followed the steps as mentioned in this blog : https://www.linkedin.com/pulse/aws-glue-data-catalog-metastore-databricks-deepak-rajak/ but when trying to saveAsTable(table_name), it is giving an error as IllegalArgumentException: Path must be ...
- 6833 Views
- 4 replies
- 3 kudos
- 3 kudos
Hey @Raghu Bharadwaj Tallapragada​ Just wanted to check in if you were able to resolve your issue or do you need more help? We'd love to hear from you.Thanks!
- 3 kudos
- 1014 Views
- 0 replies
- 1 kudos
Trouble Displaying Full Size Images from Spark Dataframe
Hi, I have followed this guide (https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/image-data-source.html) to successfully load some image data into a spark df and display it as a thumbnail. I would like to display a single image fr...
- 1014 Views
- 0 replies
- 1 kudos
- 7737 Views
- 6 replies
- 2 kudos
Resolved! How to see if condition is True / False for all rows in a DataFrame?
Assume that I have a Spark DataFrame, and I want to see if records satisfy a condition.Example dataset:# Prepare Data data = [('A', 1), \ ('A', 2), \ ('B', 3) ] # Create DataFrame columns= ['col_1', 'col_2'] df = spark.createDataF...
- 7737 Views
- 6 replies
- 2 kudos
- 2 kudos
Hi you can use display() or show() function that will provide you expected results.
- 2 kudos
- 771 Views
- 1 replies
- 1 kudos
medium.com
Sorting In Spark**How to sort null values First and last of the records in the Spark data frame?Please find the answershttps://medium.com/@sharikrishna26/sorting-in-spark-a57db245ecd4
- 771 Views
- 1 replies
- 1 kudos
- 1 kudos
Yeah this is really good post,keep it up Man
- 1 kudos
- 2113 Views
- 1 replies
- 1 kudos
SocketTimeout exception when running a display command on spark dataframe
I am using runtime 9.1LTSI have a R notebook that reads a csv into a R dataframe and does some transformations and finally is converted to spark dataframe using the createDataFrame function.after that when I call the display function on this spark da...
- 2113 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Manjusha Unnikrishnan​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon. Thanks.
- 1 kudos
- 3052 Views
- 3 replies
- 5 kudos
Getting error with spark-sftp, no such file
In a databricks cluster with Scala 2.1.1 I am trying to read a file into a spark data frame using the following code.val df = spark.read .format("com.springml.spark.sftp") .option("host", "*") .option("username", "*") .option("password", "*")...
- 3052 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Andreas P​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 5 kudos
- 3519 Views
- 2 replies
- 2 kudos
Resolved! spark sql update really slow
I tried to use Spark as much as possible but experience some regression. Hopefully to get some direction how to use it correctly.I've created a Databricks table using spark.sqlspark.sql('select * from example_view ') \ .write \ .mode('overwr...
- 3519 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, @Vincent Doe​ ,Updates are available in Delta tables, but under the hood you are updating parquet files, it means that each update needs to find the file where records are stored, then re-write the file to new version, and make new file current v...
- 2 kudos
- 920 Views
- 0 replies
- 0 kudos
Converting Rows of Spark Dataframe to List
How to convert the rows of a spark dataframe to list without using Pandas.Input Spark Dataframe :Expected Output:[['A','B','C'],['1','2','3'],['4','5','6'],['7','8','9']]
- 920 Views
- 0 replies
- 0 kudos
- 3896 Views
- 3 replies
- 6 kudos
Resolved! Unable to change the index, when writing to a Azure SQL Data Warehouse
Hi,I have some data in a spark data frame and I am trying to write it to a table in Azure SQL Data Warehouse. If I use df.write.mode(saveMode="overwrite") I get this error:com.microsoft.sqlserver.jdbc.SQLServerException: The statement failed. Column ...
- 3896 Views
- 3 replies
- 6 kudos
- 6 kudos
Thanks @Hubert Dudek​ . Do you know if there is a way to run a create table statement on Azure Synapse/Azure SQL Datawarehouse from Databricks?
- 6 kudos
- 5955 Views
- 4 replies
- 3 kudos
Resolved! Writing Spark data frame to ADLS is taking Huge time when Data Frame is of Text data.
Spark data frame with text data when schema is in Struct type spark is taking too much time to write / save / push data to ADLS or SQL Db or download as csv.
- 5955 Views
- 4 replies
- 3 kudos
- 3 kudos
@shiva Santosh​ Have to checked the count of the dataframe that you are trying to save to ADLS?As @Joseph Kambourakis​ mentioned the explode can result in 1-many rows, better to check data frame count and see if Spark OOMs in the workspace.
- 3 kudos