Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Can someone help me understand what compute the SharePoint Connector is using by default? I'm trying to use Job Compute, but it "seems" like it may be defaulting to serverless.
The managed SharePoint connector uses serverless compute; it can't be switched to classic Job Compute.
The Job you see is the orchestration layer: Lakeflow Connect creates it for the schedule and runs the ingestion pipeline as a task, while the pipeline itself stays on managed serverless compute.
Make sure serverless compute and the SharePoint Beta preview are enabled for the workspace, as required by the managed connector setup.
If managed serverless is acceptable, keep the current pipeline; there is no cluster configuration to change.
If you need to choose the compute, use the standard SharePoint connector with SQL, PySpark, or Auto Loader on DBR 17.3 LTS or later.
The managed SharePoint connector uses serverless compute; it can't be switched to classic Job Compute.
The Job you see is the orchestration layer: Lakeflow Connect creates it for the schedule and runs the ingestion pipeline as a task, while the pipeline itself stays on managed serverless compute.
Make sure serverless compute and the SharePoint Beta preview are enabled for the workspace, as required by the managed connector setup.
If managed serverless is acceptable, keep the current pipeline; there is no cluster configuration to change.
If you need to choose the compute, use the standard SharePoint connector with SQL, PySpark, or Auto Loader on DBR 17.3 LTS or later.
Thanks @anagilla , this was really helpful to point me in the right direction. One edit, while you're right on the SharePoint connector that is available/supported by databricks being serverless only at this time (not sure if in the future this will be expanded to run with job compute and/or all-purpose compute), it is actually possible to create your own "Community Connector" that runs either with Serverless toggled on (by default) or off. However, if Serverless is toggled off, this comes with some additional Pipeline environment Dependency configurations which both of these posts dive into:
