withColumnRenamed does not work with databricks-connect 14.3.0

[dbal]

I am not able to run our unit tests suite due a possible bug in the databricks-connect library.

The problem is with the Dataframe transformation withColumnRenamed. When I run it in a Databricks cluster (Databricks Runtime 14.3 LTS), the column is renamed correctly. But when we run in the local machine (databricks-connect) unit tests, the column is never renamed.

Investigating the query plan in both environments I noticed that the Physical Plan in Databricks actually generates a Projection. The plan generated by databricks-connect does not. See images below.

Running in Databricks Runtime 14.3 LTS

Running in local machine using databricks-connect:14.3.0

 

If I instead of withColumnRenamed I use select, then it works as expected, as shown below:

[shan_chandra]

@dbal - can you please try withColumnsRenamed() instead

Reference: https://docs.databricks.com/en/release-notes/dbconnect/index.html#databricks-connect-1430-python

[dbal]

Hi @shan_chandra  thank you for your answer.

I am using Scala, not Python. Nevertheless, I tried using the withColumnsRenamed(), but sadly it shows the same issue.