The problem is with the Dataframe transformation withColumnRenamed. When I run it in a Databricks cluster (Databricks Runtime 14.3 LTS), the column is renamed correctly. But when we run in the local machine (databricks-connect) unit tests, the column is never renamed.
Investigating the query plan in both environments I noticed that the Physical Plan in Databricks actually generates a Projection. The plan generated by databricks-connect does not. See images below.
Running in Databricks Runtime 14.3 LTS
Running in local machine using databricks-connect:14.3.0
If I instead of withColumnRenamed I use select, then it works as expected, as shown below: