Hi mmayorga, thanks for the answer, but as I describe above, this was working before and I am looking for a specific reason on why this isn't working anymore.

Something must've changed, and since it isn't working for you anymore, too; it probably is something from Databricks side.