Databricks

kolangareth · 01-13-2022

I am trying to do a dynamic partition overwrite on delta table using replaceWhere option. This was working fine until I upgraded the DB runtime to 9.1 LTS from 8.3.x. I am concatenating 'year', 'month' and 'day' columns and then using to_date functio...

kolangareth · 01-14-2022

Hi @Hubert Dudek , to do that I must also add this new column in the target delta table, which I don't want to do. Adding this new column only in the dataframe to write will not work because of two reasons,delta will give a schema mismacth error whi...

kolangareth · 01-14-2022

I think it is neither. replaceWhere can take expressions, because I've tried using concat without to_date and it doesn't fail. The expression isn't faulty, because it works with the spark conf setting to fallback to old behaviour and also the same ex...

kolangareth · 01-14-2022

Hi @Werner Stinckens , the to_date function works fine everywhere else like filter, withColumn etc. If you meant to create a new column in the dataframe and then use it in the replaceWhere - it's not something I want to do, because to do that I must...

Databricks

User Stats

User Activity

to_date not functioning as expected after introduction of arbitrary replaceWhere in Databricks 9.1 LTS

Re: to_date not functioning as expected after introduction of arbitrary replaceWhere in Databricks 9.1 LTS

Re: to_date not functioning as expected after introduction of arbitrary replaceWhere in Databricks 9.1 LTS

Re: to_date not functioning as expected after introduction of arbitrary replaceWhere in Databricks 9.1 LTS