Databricks Community

JacintoArias · 01-28-2022

Hi,While developing an ETL for a large dataset I want to get a sample of the top rows to check that my the pipeline "just runs", so I add a limit clause when reading the dataset.I'm surprised to see that instead of creating a single task as in a sho...

JacintoArias · 06-12-2024

We had a similar situation, @Hubert-Dudek we are using delta, but we are having some problems when propagating updates via merge, as you cannot read the resulting table as streaming source anymore... so using complete overwrite over parquet partition...

JacintoArias · 02-01-2022

Thank you very much KanizI know understand that Predicate Pushdown only works for WHERE clauses with partitions and the internal statistics of parquet files.I would be interesting though to have a nice way to retrieve just a few rows of a highly part...

JacintoArias · 01-28-2022

Hi Hubert, thanks for the quick response,> Just point it to folder with one partition.This will work but I'd like a general solution than can be applied without changing the path or having to look inside the folder architecture, some times you don't ...

Databricks Community

User Stats

User Activity

Spark predicate pushdown on parquet files when using limit

Re: Dynamic partition overwrite with Streaming Data

Re: Spark predicate pushdown on parquet files when using limit

Re: Spark predicate pushdown on parquet files when using limit