Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-09-2025 11:26 PM
I am able to perform the below operation for a delta table.
SELECT *,_metadata.file_name FROM anytable where condition.
https://docs.databricks.com/en/ingestion/file-metadata-column.html
You can use something like
df = spark.read \ .format("json") \ .schema(schema) \ .load("dbfs:/tmp/*") \ .select("*", "_metadata") display(df)
. You can enable the rescued data column by setting the option rescuedDataColumn to a column name, such as _rescued_data with spark.read.option("rescuedDataColumn", "_rescued_data").format("json").load(<path>).
https://docs.databricks.com/en/query/formats/json.html#rescued-data-column