I am trying to build a dlt pipeline which fetches data from fabric onelake tables and stores it in databricks.

To do this, i am using azure service principals and  databricks asset bundles. i have setup authentication configs as per official documentaion from azure. but i am facing below error even though i have all the necessary acceses

SparkException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501

the code works perfectly fine when i run them in personal compute but throws this error when i run them in job clusters.