Databricks Community

RobDineen

Hi there i have a simple Pyspark To_date function but fails due to days or months from 1-9 sois there a nice easy way to get round this at allRegardsRob

RobDineen

I have the following Spark (Save As Table) example. sometimes it works fine, sometimes it failsCode below with file listed in "/temp" directory. This has worked fine as it is, but when I have to create a new Cluster, as I am using the community edit...

RobDineen

Hello there, i have a temp table where i want to remove a null / empty values ( see below )if there are no rows to delete, then shouldn't it just say Zero rows affected?

RobDineen · 10-11-2024

I am just working with Databricks, and have come across an issue where delta tables have been created in the catalogue but do not actually exist. see screenshot for script ive been running and error messagesis this a bug, or am i missing something o...

RobDineen

Resolved using format_string dff = df.withColumn("DayofMonthFormatted", when(df.DayofMonth.isin([1,2,3,4,5,6,7,8,9]), format_string("0%d", df.DayofMonth)).otherwise(df.DayofMonth))

RobDineen

Hi @VZLA any idea with the below work around, I'm nearly there.

RobDineen

i have been trying to solve it with the following New column on the fly, if DayofMonth in (1,2,3,4,5,6,7,8,9) then put a 0 before, else leave as is.obviously I'm trying to insert the 0 incorrectly. but wondering how? nearly there

RobDineen

This seems to work, along with explicitly dropping the Database, and re running all code within Notebook.Thank you

RobDineen

I have updated the location so it is explicit. but if I use mode("overwrite") that fails as well.

Databricks Community

User Stats

User Activity

Pyspark to_date not coping with single digit Day or Month

saveAsTable sometimes works sometimes dont

%SQL delete from temp table driving me mad

Delta table in catalogue are showing but DO NOT exist

Re: Pyspark to_date not coping with single digit Day or Month

Re: Pyspark to_date not coping with single digit Day or Month

Re: Pyspark to_date not coping with single digit Day or Month

Re: saveAsTable sometimes works sometimes dont

Re: saveAsTable sometimes works sometimes dont