cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
farbodr
by • New Contributor II
  • 4404 Views
  • 5 replies
  • 1 kudos

Shapley Progressbar

The shapley progress bar or tqdm progress bar in general doesn't show in notebooks. Do I need to set something special to get this or any other similar widgets to work?

Data Engineering
Reply
  • 4404 Views
  • 5 replies
  • 1 kudos
View Replies
Latest Reply
richk7
New Contributor II
  • 1 kudos

I think you're looking for tqdm.notebookfrom time import sleepfrom tqdm.notebook import tqdmfor _ in tqdm(range(20)): sleep(5)

  • 1 kudos
4 More Replies
amitdatabricksc
by • New Contributor II
  • 9597 Views
  • 4 replies
  • 2 kudos

how to zip a dataframe

how to zip a dataframe so that i get a zipped csv output file. please share command. it is only 1 dataframe involved and not multiple. 

Data Engineering
Reply
  • 9597 Views
  • 4 replies
  • 2 kudos
View Replies
Latest Reply
-werners-
Esteemed Contributor III
  • 2 kudos

writing to a local directory does not work.See this topic:https://community.databricks.com/s/feed/0D53f00001M7hNlCAJ

  • 2 kudos
3 More Replies
Michelle_-_Devp
by • New Contributor III
  • 1037 Views
  • 1 replies
  • 1 kudos

Resolved! How is brainstorming going?

Wondering if anyone is willing to share their project ideas here. It would be great to know how things are going and if anyone has a good open-source dataset they are willing to share.

Data Engineering
Reply
  • 1037 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
bayang
New Contributor III
  • 1 kudos

Good, read their docs to get a lot of info to sharpen this hackathon

  • 1 kudos
amitca71
by • Contributor II
  • 2775 Views
  • 1 replies
  • 1 kudos

Resolved! sedona/shapely error Unknown WKB type 16

Hi,i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.once i try to read it ei...

Data Engineering
Reply
  • 2775 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
<div style= This widget could not be displayed.
" src="" />
This widget could not be displayed.
This widget could not be displayed.
This widget could not be displayed.
  • 1 kudos

This widget could not be displayed.
Hi,i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.once i try to read it ei...

This widget could not be displayed.
  • 1 kudos
This widget could not be displayed.
Bartek
by • Contributor
  • 3176 Views
  • 1 replies
  • 1 kudos

Save Spark DataFrame to shape file (.shp format)

Hello,I know how to create .shp file from Geopandas dataframe using code similar to this, also mentioned on SO:gpd_df = geopandas.GeoDataFrame(pandas_df, geometry='geom') gpd_df .to_file("username/nh.shp")However I have .parquet files that I can load...

Data Engineering
Reply
  • 3176 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Bartosz Maciejewski​ :Spark does not have native support for writing Shapefiles directly. However, you can use a third-party library such as GeoPandas or PyShp to write your Spark DataFrame to a Shapefile.Here's an example of how to use GeoPandas to...

  • 1 kudos
Mohan_Potla
by • New Contributor
  • 1572 Views
  • 2 replies
  • 0 kudos

Link to "Lakehouse Fundamentals Accreditation (V2)" in Databricks Academy

Please share link to take Lakehouse Fundamentals Accreditation (V2) in Databricks Academy.

Data Engineering
Reply
  • 1572 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Adding @Vidula Khanna​ for visibility.

  • 0 kudos
1 More Replies
theSoyf
by • New Contributor II
  • 4091 Views
  • 2 replies
  • 1 kudos

How to write to Salesforce object using Spark Salesforce Library

Hi I'm facing an issue when writing to a salesforce object. I'm using the springml/spark-salesforce library. I have the above libraries installed as recommended based on my research.I try to write like this:(_sqldf .write .format("com.springml.spar...

Screen Shot 2022-12-14 at 8.18.07 AM
Data Engineering
Reply
  • 4091 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Gauthy
New Contributor II
  • 1 kudos

Im facing the same issue while trying to write to Salesforce, if you have found a resolution could you please share it ?

  • 1 kudos
1 More Replies
Vijaykumarj
by • New Contributor III
  • 4741 Views
  • 4 replies
  • 3 kudos

Generate sh2 hashkey while loading files to delta table

I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...

image.png
Data Engineering
Reply
  • 4741 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 3 kudos

Hi , Could you please provide the error code?

  • 3 kudos
3 More Replies
siva_thiru
by • Contributor
  • 1040 Views
  • 0 replies
  • 6 kudos

Happy to share that #WAVICLE​  was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals wi...

Happy to share that #WAVICLE​  was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals with KCT College, Coimbatore, India.

Workshop Standee
Data Engineering
Reply
  • 1040 Views
  • 0 replies
  • 6 kudos
jimnaik
by • New Contributor III
  • 19848 Views
  • 2 replies
  • 1 kudos

Resolved! How to execute .sh and .py file in the workspace?

I want to execute shell script which is running .py file. May I know how to run .sh file and .py files in Databricks workspace?

Data Engineering
Reply
  • 19848 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
jimnaik
New Contributor III
  • 1 kudos

I tried executing like this and it worked: %sh /dbfs/***/***/***.sh

  • 1 kudos
1 More Replies
brickster_2018
by • Databricks Employee
  • 1630 Views
  • 1 replies
  • 0 kudos

Are there some shard-level setting to make every cluster in that shard use KMS encryption when writing to S3

Data Engineering
Reply
  • 1630 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
aladda
Databricks Employee
  • 0 kudos

You could potentially do this through a Global Init Script - https://docs.databricks.com/clusters/init-scripts.html

  • 0 kudos
ArielHerrera
by • New Contributor II
  • 17399 Views
  • 5 replies
  • 2 kudos

Resolved! How to display SHAP plots?

I am looking to display SHAP plots, here is the code:import xgboost import shap shap.initjs() # load JS visualization code to notebookX,y = shap.datasets.boston() # train XGBoost model model = xgboost.train({"learning_rate": 0.01}, xgboost.DMatri...

0693f000007OoIfAAK
Data Engineering
Reply
  • 17399 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
lrnzcig
New Contributor II
  • 2 kudos

As @Vinh dqvinh87​  noted, the accepted solution only works for force_plot. For other plots, the following trick works for me:import matplotlib.pyplot as plt p = shap.summary_plot(shap_values, test_df, show=False) display(p)

  • 2 kudos
4 More Replies
Labels