topic Re: withColumnRenamed does not work with databricks-connect 14.3.0 in Data Engineering

withColumnRenamed does not work with databricks-connect 14.3.0

dbal — Fri, 10 May 2024 23:22:30 GMT

I am not able to run our unit tests suite due a possible bug in the databricks-connect library.

The problem is with the Dataframe transformation withColumnRenamed. When I run it in a Databricks cluster (Databricks Runtime 14.3 LTS), the column is renamed correctly. But when we run in the local machine (databricks-connect) unit tests, the column is never renamed.

Investigating the query plan in both environments I noticed that the Physical Plan in Databricks actually generates a Projection. The plan generated by databricks-connect does not. See images below.

Running in Databricks Runtime 14.3 LTS

Running in local machine using databricks-connect:14.3.0

If I instead of withColumnRenamed I use select, then it works as expected, as shown below:

Re: withColumnRenamed does not work with databricks-connect 14.3.0

shan_chandra — Thu, 16 May 2024 16:11:11 GMT

@dbal - can you please try withColumnsRenamed() instead

Reference: https://docs.databricks.com/en/release-notes/dbconnect/index.html#databricks-connect-1430-python

Re: withColumnRenamed does not work with databricks-connect 14.3.0

dbal — Fri, 17 May 2024 13:56:44 GMT

Hi @shan_chandra thank you for your answer.

I am using Scala, not Python. Nevertheless, I tried using the withColumnsRenamed(), but sadly it shows the same issue.