how to zip a dataframe so that i get a zipped csv output file. please share command. it is only 1 dataframe involved and not multiple.
writing to a local directory does not work.See this topic:https://community.databricks.com/s/feed/0D53f00001M7hNlCAJ
Wondering if anyone is willing to share their project ideas here. It would be great to know how things are going and if anyone has a good open-source dataset they are willing to share.
Good, read their docs to get a lot of info to sharpen this hackathon
Hi,i stream data from postgis to s3 using debezium. postgis->debezium->s3->spark(databricks)once read it i decode it and i can see that the binary representation is similiar to what i have in postgis, on a wkb formated column.once i try to read it ei...
Hi @Amit Cahanovich​, The error message "Unknown WKB type 16" indicates that the WKB data you are trying to read has a geometry type that the library does not recognize. WKB type 16 is not valid in the Simple Feature Access (SFA) standard, the most w...
Hello,I know how to create .shp file from Geopandas dataframe using code similar to this, also mentioned on SO:gpd_df = geopandas.GeoDataFrame(pandas_df, geometry='geom') gpd_df .to_file("username/nh.shp")However I have .parquet files that I can load...
@Bartosz Maciejewski​ :Spark does not have native support for writing Shapefiles directly. However, you can use a third-party library such as GeoPandas or PyShp to write your Spark DataFrame to a Shapefile.Here's an example of how to use GeoPandas to...
Please share link to take Lakehouse Fundamentals Accreditation (V2) in Databricks Academy.
Adding @Vidula Khanna​ for visibility.
Hi I'm facing an issue when writing to a salesforce object. I'm using the springml/spark-salesforce library. I have the above libraries installed as recommended based on my research.I try to write like this:(_sqldf .write .format("com.springml.spar...
Im facing the same issue while trying to write to Salesforce, if you have found a resolution could you please share it ?
I have files in azure data lake. I am using autoloader to read the incremental filesfiles don't have primary key to load, In this case i want to use some columns and generate an hashkey and use it as primary key to do changes.In this case i want to ...
Hi @Vijay Kumar J​(Customer)​ , We haven’t heard from you since the last response from @Debayan Mukherjee​ and @Jordan Fox​ and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please do share that with the...
The shapley progress bar or tqdm progress bar in general doesn't show in notebooks. Do I need to set something special to get this or any other similar widgets to work?
Hi @Fred Rahmanian​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
Happy to share that #WAVICLE​ was able to do a hands-on workshop on #[Databricks notebook]​ #[Databricks SQL]​ #[Databricks cluster]​ Fundamentals with KCT College, Coimbatore, India.
I want to execute shell script which is running .py file. May I know how to run .sh file and .py files in Databricks workspace?
I tried executing like this and it worked: %sh /dbfs/***/***/***.sh
You could potentially do this through a Global Init Script - https://docs.databricks.com/clusters/init-scripts.html
I am looking to display SHAP plots, here is the code:import xgboost import shap shap.initjs() # load JS visualization code to notebookX,y = shap.datasets.boston() # train XGBoost model model = xgboost.train({"learning_rate": 0.01}, xgboost.DMatri...
As @Vinh dqvinh87​ noted, the accepted solution only works for force_plot. For other plots, the following trick works for me:import matplotlib.pyplot as plt p = shap.summary_plot(shap_values, test_df, show=False) display(p)
