- 4693 Views
- 5 replies
- 2 kudos
High cost of storage when using structured streaming
Hi there, I read data from Azure Event Hub and after manipulating with data I write the dataframe back to Event Hub (I use this connector for that): #read data df = (spark.readStream .format("eventhubs") .options(**ehConf) ...
- 4693 Views
- 5 replies
- 2 kudos
- 2 kudos
I had the same problem when starting with databricks. As outlined above, it is the shuffle partitions setting that results in number of files equal to number of partitions. Thus, you are writing low data volume but get taxed on the amount of write (a...
- 2 kudos
- 858 Views
- 0 replies
- 5 kudos
Some Tips & Tricks for Optimizing costs and performance (Clusters and Ganglia):Â [Note: This list is not exhaustive]Â Leverage the DataFrame or Spar...
Some Tips & Tricks for Optimizing costs and performance (Clusters and Ganglia):[Note: This list is not exhaustive]Leverage the DataFrame or SparkSQL API’s first. They use the same execution process resulting in parity in performance but they also com...
- 858 Views
- 0 replies
- 5 kudos