According to me, your NSG config is fine. The rule 65001 on your outbound should allow your cluster to access internet. Not sure it will help, but there is this line in your log: 'Error code: ERROR_NO_MATCHING_DISTRIBUTION'
I don't know exactly how you are trying to install your python libs?
I ask it to you me because your issue makes me think about a problem I had when I was trying to install some python wheels on my cluster. I was in a different NSG configuration that you. For security reason my cluster doesn't have access to the internet, so I have to install python wheels already uploaded on the cluster with the help of a init script.
At a time, I had this "no matching distribution" error when the wheels I was trying to install was not in sync with the cluster. I had previously downloaded the wheels files from a cluster with Databricks Runtime 15.4, then uploaded them to a cluster with Runtime 12.2 and then trying to install with pip.
