Fastest way to get data into PowerBI

[sondrewb]

Hello. Databricks can be connected with PowerBi to visualize data. The process is described here. Of the various approaches described there, what is the fastest for large amounts of data? Are there even any difference in performance for the different approaches? Are there other approaches than the ones mentioned above for importing large amounts of data into BI tools?

Thanks.

[Hubert-Dudek]

SQL endpoint. Question How big is the dataset? As usually limitation is dataset size on the PowerBI as there is a limit of 1GB for pro and 10GB for premium. 10GB for Spark is a small dataset but large for PowerBI.

[sondrewb]

@Hubert Dudek​ , Thanks for the quick response. Could you point me to an overview/comparison of the performance of the different ways to connect? Why is SQL Endpoint faster?

[Anonymous · Accepted solution]

Hi @sondrewb​ you can connect BI tools to Databricks SQL endpoints to query data in tables through an ODBC/JDBC protocol integrated in our Simba drivers. With Cloud Fetch, which we released in Databricks Runtime 8.3 and Simba ODBC 2.6.17 driver, we introduce a new mechanism for fetching data in parallel via cloud storage such as AWS S3 and Azure Data Lake Storage to bring the data faster to BI tools. In our experiments using Cloud Fetch, we observed a 10x speed-up in extract performance due to parallelism.

Please find the below document reference

https://databricks.com/blog/2021/08/11/how-we-achieved-high-bandwidth-connectivity-with-bi-tools.htm...

[jose_gonzalez]

Hi @Soner Candan​,

Just a friendly follow-up. Do you still need help or our community responses helped? Please let us know.