Okay so I resolved this problem by creating a VPC peering between the Databricks VPC and my local VPC that hosts my Postgresql instance. https://docs.databricks.com/administration-guide/cloud-configurations/aws/vpc-peering.html?_ga=2.423....

I have now been able to create a dataframe linking to my Postgresql RDS with the same notebook python code as seen above.