@shan_chandra I was able to install and rerun the tests and I am able to see 70k avg process/sec - and job is not crashing any longer. My goal is to see if I can achieve 500k msgs/sec. I will continue testing next week. 

Will databricks runtime update it's package with the library we needed to install manually - what will be ETA?

Thanks so much for your help.