Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-28-2022 08:36 AM
Hi Hubert, thanks for the quick response,
> Just point it to folder with one partition.
This will work but I'd like a general solution than can be applied without changing the path or having to look inside the folder architecture, some times you don't have access easily to S3 or other filesystems (when using a catalog)
> Additionally specify schema to avoid inferSchema behavior.
This didn't work for me, at least using parquet files, it still creates the huge amount of tasks.
> you can use samplingRatio=None
didn't work for me either, it seems it still reads metadata/rows from parquet files