Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
While returning results to PowerBI, Databricks Completed the session (in 9 mins) but PowerBI waiting for the results (more than 7 hrs. for 20 GB of data).Could you please help us on this.
@Janga Reddy It's really hard to find a solution here without further investigating PBI Gateway performance/Data models etc.If Databricks completed the session in 9 mins then I assume that the issue could be with the performance of PBI datasets.
Process is to pickup data from an external table in Azure Synapse and combine with other tables and write to another synapse tables. Data for external table is in Azure storage. It was fine for months ,all of sudden last week it errored out with err...
Also do check out this blog from this stack overflow question.https://stackoverflow.com/questions/66747544/databricks-write-back-to-azure-synapse-errorhttps://chinnychukwudozie.com/2020/11/13/write-data-from-azure-databricks-to-azure-synapse-analytic...
Hi @Krishna Zanwar Does @Debayan Mukherjee response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
Hi, @Krishna Zanwar Could you please raise a support case to report the bug. Please refer https://docs.databricks.com/resources/support.html to engage with Databricks Support.
I hv a complex script which consuming more then 100GB data and have some aggregation on it and in the end I am simply try simply write/display data from Data frame. Then i am getting issue (assertion failed: Invalid shuffle partition specs: ).Pls hel...
