I am not able to run our unit tests suite due a possible bug in the databricks-connect library.
The problem is with the Dataframe transformation withColumnRenamed. When I run it in a Databricks cluster (Databricks Runtime 14.3 LTS), the column is renamed correctly. But when we run in the local machine (databricks-connect) unit tests, the column is never renamed.
Investigating the query plan in both environments I noticed that the Physical Plan in Databricks actually generates a Projection. The plan generated by databricks-connect does not. See images below.
Running in Databricks Runtime 14.3 LTS
Running in local machine using databricks-connect:14.3.0
If I instead of withColumnRenamed I use select, then it works as expected, as shown below:
Hi @shan_chandra thank you for your answer.
I am using Scala, not Python. Nevertheless, I tried using the withColumnsRenamed(), but sadly it shows the same issue.
