You can try to use BULK INSERT.https://learn.microsoft.com/en-us/sql/t-sql/statements/bulk-insert-transact-sql?view=sql-server-ver16Also using Data Factory instead of Databricks for the copy can be helpful.
I am not familiar with st_intersects, but it seems that it runs solely on the driver (as python code, not spark).Does mosaic work in pyspark?If not: try to use a larger driver.
Sql Serverless works on external tables. We only use external tables with serverless.But no Unity (yet) and not installed on a private network.I think the latter could be the issue as with the serverless option, the actual VMs are not installed with...
checked it with my mount script and that is exactly the same except that I do not put a '/' after dfs.core.windows.netYou might wanna try that.Also, is Unity enabled? Because Unity does not allow mounts.
