If I'm understanding, all you really want to do is have a pre/post - process function running with your model, is that correct? If so, you can do this by using the MLflow pyfunc model. Something like they do here:https://docs.databricks.com/en/machi...
Well, Autoloader could work nicely with the notification event for arriving files. You could probably specify a window duration for your "on-time" arrivels and that could be your base check for on time. As files arrive they go to their window and whe...
I would guess that at this point, you will have to destroy them manually. In the future, you might want to let your TF script create all the requisite network resources; then the script will also be able to destroy them. This is a general GKE issue a...
