cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Etyr
by Contributor
  • 5961 Views
  • 4 replies
  • 2 kudos

Resolved! slow Fetching results by client in databricks SQL calling from Azure Compute Instance (AML)

I'm using `databricks-sql-connector` in python3.8 to connect to an Azure SQL Wharehouse inside an Azure Machine Learning Compute Instance.I have this large result query, looking at the `query history` I check the time spent on doing the query, and se...

first_time_query
  • 5961 Views
  • 4 replies
  • 2 kudos
Latest Reply
Etyr
Contributor
  • 2 kudos

So I made some few tests. Since you said that the Databricks SQL driver wasn't made to retrieve that amount of data. I went on Spark.I fired up a small spark cluster, the query was as fast as on SQL Warehouse, then I did a df.write.parquet("/my_path/...

  • 2 kudos
3 More Replies
Labels