Databricks

Bartek · 01-27-2023

Hello,I know how to create .shp file from Geopandas dataframe using code similar to this, also mentioned on SO:gpd_df = geopandas.GeoDataFrame(pandas_df, geometry='geom') gpd_df .to_file("username/nh.shp")However I have .parquet files that I can load...

Bartek · 01-19-2023

I want to trigger Databricks job from Airflow using DatabricksSubmitRunDeferrableOperator and I need to pass configuration params. Here is excerpt from my code (definition is not complete, only crucial properties):from airflow.providers.databricks.op...

Bartek · 12-14-2022

About 2 weeks ago I started course on "Optimizing Apache Spark on Databricks" from official Databricks academy. It is heavily based on Spark UI simulator experiments that were available here: https://www.databricks.training/spark-ui-simulator and for...

Bartek · 11-26-2022

I am learning how to optimize Spark applications with experiments from Spark UI Simulator. There is experiment #1596 about data skew and in command 2 there is comment about how many partitions will be set as default:// Factor of 8 cores and greater ...

Bartek · 12-15-2022

Hi @Landan George,thanks, yes, it works with this protocol.I was not aware that the certificate will be dropped - but for me the most important that the resource is available.CheersBartek

Bartek · 12-14-2022

one more solution - you can read .zip using old good pandas `read_csv` method (https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html#pandas.read_csv)import pandas as pd simple_csv_zipped = pd.read_csv("/dbfs/FileStore/simple_file.zip")s...

Bartek · 12-14-2022

Great you pointed out @Ben Elbert that spark allows to read compressed files (`compression` property mentioned here: https://spark.apache.org/docs/latest/sql-data-sources-csv.html). Still, it won't work with .zip archive.

Bartek · 12-13-2022

Here is the instruction how to connect with storage account: https://learn.microsoft.com/en-us/azure/databricks/external-data/azure-storage.Setting it on the cluster level: https://docs.databricks.com/clusters/configure.html#spark-configuration.Examp...

Bartek · 12-13-2022

Additionally, if you don't want or can't unzip whole archive, you can list the contents of the archive and unzip only selected file.Still, as @Joseph Kambourakis asked - why can't you just unzip it? What's blocking you?

Databricks

User Stats

User Activity

Save Spark DataFrame to shape file (.shp format)

How to pass all dag_run.conf parameters to python_wheel_task

Spark UI simulator is not available online

Number of partitions in Spark UI Simulator experiment

Re: Spark UI simulator is not available online

Re: I have to read zipped csv file using spark without unzipping it. can anyone please provide pyspark/spark sql code for that?

Re: I have to read zipped csv file using spark without unzipping it. can anyone please provide pyspark/spark sql code for that?

Re: External Table issue format in databricks

Re: I have to read zipped csv file using spark without unzipping it. can anyone please provide pyspark/spark sql code for that?