Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-17-2022 08:14 AM
Seems that file is corrupted maybe you can ignore them by setting:
spark.conf.set("spark.sql.files.ignoreCorruptFiles", "true")
you can also check that setting:
sqlContext.setConf("spark.sql.parquet.filterPushdown","false")
you can register your files as table (pointed to that location with files) with correct schema set and than try to run:
%sql
MSCK REPAIR TABLE table_name
https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-ddl-repair-table.html
My blog: https://databrickster.medium.com/