datahero
New Contributor II

I had the similar issue with the Unity Catalog updrage, found the following solution working, based on the documentation - 

https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/functions/input_file_name

 .selectExpr("*", "_metadata as source_metadata")
  # .withColumn('file_path', input_file_name())