- 17412 Views
- 5 replies
- 2 kudos
Resolved! How to display SHAP plots?
I am looking to display SHAP plots, here is the code:import xgboost import shap shap.initjs() # load JS visualization code to notebookX,y = shap.datasets.boston() # train XGBoost model model = xgboost.train({"learning_rate": 0.01}, xgboost.DMatri...
- 17412 Views
- 5 replies
- 2 kudos
- 2 kudos
As @Vinh dqvinh87​ noted, the accepted solution only works for force_plot. For other plots, the following trick works for me:import matplotlib.pyplot as plt p = shap.summary_plot(shap_values, test_df, show=False) display(p)
- 2 kudos
- 13257 Views
- 3 replies
- 0 kudos
KNN classifier on Spark
Hi Team , Can you please help me in implementing KNN classifer in pyspark using distributed architecture and processing the dataset. Even I want to validate the KNN model with the testing dataset. I tried to use scikit learn but the program is runn...
- 13257 Views
- 3 replies
- 0 kudos
- 0 kudos
Hey, about about using NEC Frovedis (https://github.com/frovedis/frovedis) framework for the same. Refer: https://github.com/frovedis/frovedis/blob/master/src/foreign_if/python/examples/unsupervised_knn_demo.py It works on a distributed framework (...
- 0 kudos
- 22038 Views
- 1 replies
- 0 kudos
How can I read all the files in a folder on S3 into several pandas dataframes?
import pandas as pd import glob path = "s3://somewhere/" # use your path all_files = glob.glob(path + "/*.csv") print(all_files) li = [] for filename in all_files: dfi = pd.read_csv(filename,names =['acct_id', 'SOR_ID'], dtype={'acct_id':str,...
- 22038 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @zhaoxuan210, Please go through the below answer,https://stackoverflow.com/questions/52855221/reading-multiple-csv-files-from-s3-bucket-with-boto3
- 0 kudos
- 5383 Views
- 6 replies
- 0 kudos
- 5383 Views
- 6 replies
- 0 kudos
- 0 kudos
Hello, What is most efficient between RDD and DataFrame ? (I mean better to cache, consume less memory) Thanks you,
- 0 kudos
- 35657 Views
- 5 replies
- 3 kudos
What's the difference between Interactive Clusters and Job Cluster?
I am new to databricks. I would like to know what is the difference between Interactive Clusters and Job Cluster? There are no official document now.
- 35657 Views
- 5 replies
- 3 kudos
- 3 kudos
Sports news Football news International football news Football news Thai football news, Thai football Follow news, know sports news at Siamsportnews
- 3 kudos
- 21556 Views
- 9 replies
- 1 kudos
How can I change the parquet compression algorithm from gzip to something else?
Spark, by default, uses gzip to store parquet files. I would like to change the compression algorithm from gzip to snappy or lz4.
- 21556 Views
- 9 replies
- 1 kudos
- 9231 Views
- 1 replies
- 0 kudos
convert string dataframe column MM/dd/yyyy hh:mm:ss AM/PM to timestamp MM-dd-yyyy hh:mm:ss
How to convert string 6/3/2019 5:06:00 AM to timestamp in 24 hour format MM-dd-yyyy hh:mm:ss in python spark.
- 9231 Views
- 1 replies
- 0 kudos
- 0 kudos
You would use a combination of the functions: pyspark.sql.functions.from_unixtime(timestamp, format='yyyy-MM-dd HH:mm:ss') (documentation) and pyspark.sql.functions.unix_timestamp(timestamp=None, format='yyyy-MM-dd HH:mm:ss') (documentation)from pysp...
- 0 kudos
- 7123 Views
- 1 replies
- 0 kudos
How to deal with column name with .(dot) in pyspark dataframe??
We are streaming data from kafka source with json but in some column we are getting .(dot) in column names.streaming json data: df1 = df.selectExpr("CAST(value AS STRING)") {"pNum":"A14","from":"telecom","payload":{"TARGET":"1","COUNTRY":"India"...
- 7123 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mithu Wagh you can use backticks to enclose the column name.df.select("`col0.1`")
- 0 kudos
- 6265 Views
- 2 replies
- 0 kudos
Resolved! Saving to parquet with SaveMode.Overwrite throws exception
Hello, I'm trying to save DataFrame in parquet with SaveMode.Overwrite with no success. I minimized the code and reproduced the issue with the following two cells: > case class MyClass(val fld1: Integer, val fld2: Integer) > > val lst1 = sc.paralle...
- 6265 Views
- 2 replies
- 0 kudos
- 0 kudos
results.select("ValidationTable", "Results","Description","CreatedBy","ModifiedBy","CreatedDate","ModifiedDate").write.mode('overwrite').save("
- 0 kudos
- 4930 Views
- 3 replies
- 0 kudos
How to specify multiple files in --py-files in spark-submit command for databricks job? All the files to be specified in --py-files present in dbfs: .
I have created a databricks in azure. I have created a cluster for python 3. I am creating a job using spark-submit parameters. How to specify multiple files in --py-files in spark-submit command for databricks job? All the files to be specified in ...
- 4930 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Nandha Kumar,please go through the below docs to pass python files as job,https://docs.databricks.com/dev-tools/api/latest/jobs.html#sparkpythontask
- 0 kudos
- 4139 Views
- 4 replies
- 0 kudos
- 4139 Views
- 4 replies
- 0 kudos
- 0 kudos
@cfregly​ : For DataFrames, you can use the following code for using groupBy without aggregations.Df.groupBy(Df["column_name"]).agg({})
- 0 kudos
- 5862 Views
- 1 replies
- 0 kudos
How to read binary data in pyspark
I'm reading binary file http://snap.stanford.edu/data/amazon/productGraph/image_features/image_features.b using pyspark.from io importStringIO import array img_embedding_file = sc.binaryRecords("s3://bucket/image_features.b",4106)def mapper(featur...
- 5862 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @tourist_on_road, please go through the below spark docs,https://spark.apache.org/docs/2.3.0/api/python/pyspark.html#pyspark.SparkContext.binaryFiles
- 0 kudos
- 14081 Views
- 1 replies
- 0 kudos
Resolved! SQL variables in a notebook
Hi, In an SQL notebook, using this link: https://docs.databricks.com/spark/latest/spark-sql/language-manual/set.html I managed to figure out to set values and how to get the value. SET my_val=10; //saves the value 10 for key my_val SET my_val; //dis...
- 14081 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Mike K.., you can do this with widgets and getArgument. Here's a small example of what that might look like: https://community.databricks.com/s/feed/0D53f00001HKHZfCAP
- 0 kudos
- 32878 Views
- 12 replies
- 0 kudos
Resolved! Is there a better method to join two dataframes and not have a duplicated column?
I would like to keep only one of the columns used to join the dataframes. Using select() after the join does not seem straight forward because the real data may have many columns or the column names may not be known. A simple example belowllist = [(...
- 32878 Views
- 12 replies
- 0 kudos
- 0 kudos
This is an expected behavior. DataFrame.join method is equivalent to SQL join like thisSELECT*FROM a JOIN b ON joinExprsIf you want to ignore duplicate columns just drop them or select columns of interest afterwards. If you want to disambiguate you c...
- 0 kudos
- 14841 Views
- 2 replies
- 0 kudos
How to loop over spark dataframe with scala ?
Hello ! I 'm rookie to spark scala, here is my problem : tk's in advance for your help my input dataframe looks like this : index bucket time ap station rssi 0 1 00:00 1 1 -84.0 1 1 00:00 1 3 -67.0 2 1 00:00 1 4 -82.0 3 1 00:00 1 2 -68.0 4 1 00:00...
- 14841 Views
- 2 replies
- 0 kudos
- 0 kudos
Looping is not always necessary, I always use this foreach method, something like the following: aps.collect().foreach(row => <do something>)
- 0 kudos