without having the actual raw data, it is hard to figure out where the issue resides.  It could be the code or it could be event hub.
I'd store the raw event hub data in a data lake and use autoloader for further processing.

One thing in your code that caught my attention is the use of dropduplicates and distinct.
Are you sure you are not dropping too much?