Hi Pal,
I am experimenting with the same points you mentioned.
1, 3 - I have a 100% Azure Databricks RAG solution that processes new files coming from a frontend. Still trying to assess the costs and infrastructure as it scales. 
4 - I have been using Databricks API a lot with SP with PAT. Use it to upload files to be processed, so write permissions on volume, and to query the model - can query permissions on the serving endpoint. How is your SP getting access to files on the workspace root folder?