Databricks

JohnJustus · 11-02-2023

Hi,I am trying to access excel file that is stored in Azure Blob storage via Databricks.In my understanding, it is not possible to access using Pyspark. So accessing through Pandas is the option,Here is my code.%pip install openpyxlimport pandas as p...

JohnJustus · 09-28-2023

All,I am using Azure Databricks and at times I refer to pyspark API's to interact with data in Azure datalake using python, SQL here https://spark.apache.org/docs/3.5.0/api/python/reference/pyspark.sql/index.htmlDoes databricks website has the list o...

JohnJustus · 09-26-2023

All,I have the following code.df_Warehouse_Utilization = ( spark.table("hive_metastore.dev_ork.bin_item_detail") .join(df_DIM_Bins,col('bin_tag')==df_DIM_Bins.BinKey,'right') .groupby(col('BinKey')) .agg(count_distinct(when(col('serial_lo...

JohnJustus · 09-21-2023

Hi All,Can some one please help me with the error.This is my small python code.binmaster = binmasterdf.withColumnRenamed("tag_number","BinKey")\.withColumn ("Category", when (length("Bin")==4,'SMALL LOT'),(length("Bin")==7,'RACKING'))TypeError : with...

JohnJustus · 11-03-2023

Thank you Kaniz for the response.1. Completed2. This is completed as part of our initial environment setup OR is this something needs to be defined before the code below?3. import pandas as pdxls = pd.ExcelFile('wasbs://mnt/dl-sync-team-dev/ADP/Absen...

JohnJustus · 10-09-2023

Thanks Kaniz.Do you have the link handy for API Reference in Databricks website?

JohnJustus · 09-29-2023

Thank you DanielSo I have my code below with back stick but still doesn't work - col('bin_type').alias(`'Bin Type'`),Can you please correct me?df_Bins_Step1 = ( spark.table("hive_metastore.dev_fivetranpoc_york.bin_master") .select(col('tag_nu...

JohnJustus · 09-26-2023

Thank you Noopur for your response.So how can I accomplish with more than 2 parameters? Are there any other function available?In this case, I have to check length multiple times and categorize them accordingly.Thanks,John

JohnJustus · 09-22-2023

Thanks for your responseActually, here is my code with when function but I still see the errorbinmaster = binmasterdf.withColumnRenamed("tag_number","BinKey")\.withColumn ("Category", when (length("Bin")==4,'SMALL LOT'), when (length("Bin")==7,'RACKI...

Databricks

User Stats

User Activity

Accessing Excel file from Databricks

Pyspark API reference

Space in Column names when writing to Hive

TypeError : withcolumn() takes 3 positional arguments but 4 were given.

Re: Accessing Excel file from Databricks

Re: Pyspark API reference

Re: Space in Column names when writing to Hive

Re: TypeError : withcolumn() takes 3 positional arguments but 4 were given.

Re: TypeError : withcolumn() takes 3 positional arguments but 4 were given.