The most reliable and efficient approach is to use Azure Data Factory (ADF) with the SharePoint Online List connector to extract data from your M365 SharePoint List and write it to Azure Data Lake Storage, then use Azure Databricks to process the data from there. This avoids service principal permission issues and gives you a low-code, scalable pipeline. If you're seeing issues with expanding complex columns (like 'User Information List'), use ADF's Data Flow or post-processing in Databricks to flatten those structures. Here's a quick start guide: ADF SharePoint List Connector.