I am trying to compute outliers using approxQuantile on a Dataframe. It works fine in a Databricks notebook, but the call doesn't work correctly when it is part of a delta live table pipeline. (This is python.) Here is the line that isn't working as part of the pipeline.

quantiles = session_agg_df.approxQuantile('duration_minutes', [0.25, 0.75], 0)

Q1 = quantiles[0] # <-- Fails as part of dlt pipeline, but works in notebook

Q3 = quantiles[1] # <-- Fails as part of dlt pipeline, but works in notebook

The pipeline fails with "IndexError: list index out of range". Again, it works just fine in a notebook on the same data.

(I am trying to look for outliers in my pipeline using a very basic +/- 1.5*IQR approach.)

Also, if anybody has any suggestions on best practices for writing and debugging dlt pipelines, I'd love to hear them!

Thanks,

Miles Porter

Lead Data Scientist

Trimble, Inc.