I think it is working now. what i was trying to do, which is not right in this case was trying to test this code directly in the databricks scala notebook with just installing libraries com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.22 & com.microsoft.azure:msal4j:1.10.1 on the cluster and it was failing becuase there were few missing compile time libraries/packages that comes with azure-event-hubs-spark and msal4j libraries.

And for scala i had to make minor syntactical changes. And it started working.

thank you very much.