Hi @Retired_mod

Thanks for your response. I was able to generate the token of the service principal following this doc, later saved it in the <Databricks Token> variable prompted when running databricks-connect configure command in terminal. And was able to run the pyspark code as expected.  

But as I see read through the docs, the databricks principals token expires in 1 hr. I have few doubts: 

1. Is there a way to increase the expiration time of the token? 
2. Where is this export variable (i.e Databricks Token) or other databricks-connect variables (like Databricks Host, Cluster ID, Org ID) stored in environment?
   As I was trying to build a script which hits the service principals APIs to get the token and later exports the token value in databricks-connect configure command (doing something similar given in this link, so that I won't have to keep generating a new token everytime)
   I tried finding the variables using: <grep -iR "Databricks Token"> and other commands, but didn't get any useful results.