We sometimes process large dataframes that contain a column of IP addresses and we need to associate an Autonomous System Number (ASN) per IP address. The ASN information is provided by MaxMind in the form of a binary data file only accessible via a Python function. We have tried using a UDF that calls the Python function; however, there are issues accessing the MaxMind binary data file. How can this be done using Databricks? Note, we can successfully do this using Panda dataframes (on the Databricks platform), but we cannot rely on Pandas at scale.
