Hi,
So i've been playing around with transformWithState over the last few weeks and have got the code to work exactly as desired.
Wrote some integration tests that execute fine within the workspace in serverless and everything, the issue is however is once it's deployed and those same integration tests try to run from the Azure CI pipeline in serverless through databricks connect, I keep getting Module not found on the location where the StatefulProcessor instance is located.
Multiple other integration tests use the same directory and load everything fine across drivers and executors (use quite a bit of foreachBatch functionality), but it seems like this might be a different case. Been trying to provision the lib as a wheel, as a editable install but nothing is working so far. All modules are found in order to initialise, but I have a feeling once the workers try to deserialise the class, they don't respect the wheel attached (the job class and the stateful processor are both defined in the same .py file.
Anyone encountered this?
