The standard SharePoint connector doesn't support folder path-based filtering well. When you use wildcards in the path itself (ABC*/files/ABC*.xlsm), the connector has to enumerate directories at the SharePoint level to resolve the patterns leading to making many API calls across 5,000 directories.

• You can use pathGlobFilter instead of path wildcards

lakeflow_connection_name = 'sharepoint_dev'
sharepoint_site_url = 'https://example.sharepoint.com/sites/example_site/docs'

excel_df = (spark.read
    .format("excel")
    .option("databricks.connection", lakeflow_connection_name)
    .option("headerRows", 1)
    .option("inferSchema", False)
    .option("dataAddress", f"{sheet_name}!{sheet_range}")
    .option("pathGlobFilter", "ABC*/files/ABC*.xlsm")  # Filter here
    .load(sharepoint_site_url)
)

pathGlobFilter filters files by name after the connector retrieves the file list and is generally more efficient than path-level wildcards

• Be more specific with paths - If you know the specific ABC directory names, query them explicitly in separate reads and union the results

target_dirs = ['ABC001', 'ABC002', 'ABC003']  # directories
dfs = []

for dir_name in target_dirs:
    path = f'https://example.sharepoint.com/sites/example_site/docs/{dir_name}/files/{dir_name}*.xlsm'
    df = spark.read.format("excel")...load(path)
    Add Append df code & use​