Databricks

Christine · 07-21-2023

Not sure if this is the right place to ask this question, so let me know if it is not. I am trying to read an xls file which containts #REF values in databricks with pyspark.pandas. When I try to read the file with "pyspark.pandas.read_excel(file_pat...

Christine · 04-26-2023

Hi,I am considering creating libraries for my databricks notebooks, and found that it is possible to import functions from modules saved in repos. Is it possible to move the .py files with the functions to Workspace/Shared and still import functions ...

Christine · 03-06-2023

Due to some problems, I need to set Py4JSecurity to false, however I get the following message: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"My cluster is in Shared access mode and a runtime version 12.1 (i...

Christine · 03-03-2023

I have been using "rdd.flatMap(lambda x:x)" for a while to create lists from columns however after I have changed the cluster to a Shared acess mode (to use unity catalog) I get the following error: py4j.security.Py4JSecurityException: Method public ...

Christine · 12-05-2022

How do I add a column to an existing delta table with SQL if the column does not already exist?I am using the following code: <%sqlALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name type; >but it prints the error: <[PARSE_SYNTAX_ERROR] Synta...

Christine · 08-02-2023

Hi Kaniz, Thank you for your reply. I have been looking more into the error, and apparently the reason why the file cannot be loaded is due to an error that pops up when I open the excel sheet "One or more invalid names were detected in this workbook...

Christine · 08-02-2023

Hi, Thank you for your reply  I have already tried spark.read.format("com.crealytics.spark.excel"), but the excel file version I am using is too old. I get the error "The supplied spreadsheet seems to be Excel 5.0/7.0 (BIFF5) format. POI only suppor...

Christine · 05-01-2023

Hi @Suteja Kanuri,Thank you very much, this worked perfectly!

Christine · 03-12-2023

@Debayan Mukherjee Thank you for the clarification! I will continue with single-user cluster.

Christine · 03-08-2023

Hi @Suteja Kanuri,In this case I am using pyspark dataframe, but I am trying to get alle values from a column in that dataframe and create a list. I am using this list to filter columns in another dataframe. (see example below):value_list = pysparkD...

Databricks

User Stats

User Activity

pyspark.pandas.read_excel(engine = xlrd) reading xls file with #REF error

Is it possible to import functions from a module in Workspace/Shared instead of Repos?

"spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for cluster

ADD COLUMN IF NOT EXISTS does not recognize "IF NOT EXIST". How do I add a column to an existing delta table with SQL if the column does not already exist?

Re: pyspark.pandas.read_excel(engine = xlrd) reading xls file with #REF error

Re: pyspark.pandas.read_excel(engine = xlrd) reading xls file with #REF error

Re: Is it possible to import functions from a module in Workspace/Shared instead of Repos?

Re: "spark.databricks.pyspark.enablePy4JSecurity is not allowed when choosing an access mode"

Re: Cannot use RDD and cannot set "spark.databricks.pyspark.enablePy4JSecurity false" for cluster