Databricks

JacintoArias · 01-28-2022

Hi,While developing an ETL for a large dataset I want to get a sample of the top rows to check that my the pipeline "just runs", so I add a limit clause when reading the dataset.I'm surprised to see that instead of creating a single task as in a sho...

JacintoArias · 02-01-2022

Thank you very much KanizI know understand that Predicate Pushdown only works for WHERE clauses with partitions and the internal statistics of parquet files.I would be interesting though to have a nice way to retrieve just a few rows of a highly part...

JacintoArias · 01-28-2022

Hi Hubert, thanks for the quick response,> Just point it to folder with one partition.This will work but I'd like a general solution than can be applied without changing the path or having to look inside the folder architecture, some times you don't ...

Databricks

User Stats

User Activity

Spark predicate pushdown on parquet files when using limit

Re: Spark predicate pushdown on parquet files when using limit

Re: Spark predicate pushdown on parquet files when using limit