I am having the same issue using serverless compute. I think the issue comes from this documentation limitations https://docs.databricks.com/aws/en/structured-streaming/foreach#behavior-changes-for-foreachbatch-in-databricks-runtime-140
Hi @Jorge3 Hope I am not too lake to answer but here is my suggestion.If you reference to the docs to consume a wheel that is in a volume you can configure your job to reference your wheel in your volume.Documentation: > https://learn.microsoft.com/e...
Hi @PabloCSD,Long story short you can watch this video where I go step by step on how to set up service principal in azure, grant permissions to workspace and generate a token to itself by doing a machine to machine authentication in the Databricks C...
