Got it. As a side note, I tried above methods, but the error persisted, hence upon reading docs again, there was this statement: You must install Python 3 on your development machine, and the minor version of your client Python installation must be the same as the minor Python version of your Databricks cluster. (Link: https://docs.databricks.com/en/dev-tools/databricks-connect-legacy.html#requirements). 

On aligning my environment python version with Databricks cluster python version, the error got resolved.