@Debayan 
I get the following message:

Use "/dbfs", not "dbfs:": The function expects a local file path. The error is caused by passing a path prefixed with "dbfs:".

Having a look here https://learn.microsoft.com/en-us/azure/databricks/dbfs/unity-catalog I read the following excerpt and I am wondering If I am missing something...like I said I am using the Unity Catalog on a shared cluster

Shared access mode combines Unity Catalog data governance with Azure Databricks legacy table ACLs. Access to data in the hive_metastore is only available to users that have permissions explicitly granted.

To interact with files directly using DBFS, you must have ANY FILE permissions granted. Because ANY FILE allows users to bypass legacy tables ACLs in the hive_metastore and access all data managed by DBFS, Databricks recommends caution when granting this privilege.

Shared access mode does not support DBFS root or mounts.