Connect to delta table from mlflow pyfunc serving endpoint

[Chibberto]

Hi, I'm creating an mlflow pyfunc serving endpoint and I would like to connect to a delta table to retrieve some information within the pyfunc. Is this possible?

I ask because I don't think that serving endpoint environment has access to spark, and we would have to instantiate a spark session which would have an amount of overhead?

[Kaniz_Fatma]

Hi @Chibberto, Yes, connecting to a delta table within the MLflow sync serving endpoint is possible.

However, as you correctly pointed out, the serving endpoint environment cannot access Spark. Therefore, you would need to instantiate a Spark session, which would have an amount of overhead. 

However, you can use the Delta Standalone library, a single-node Java library that can read from and write to Delta tables. This library provides APIs to interact with table metadata in the transaction log, implementing the Delta Transaction Log Protocol to achieve the transactional guarantees of the Delta format.