I forgot to mention that on the df creation I am using filter method, cause actually p is an object:

{cntr_id : 12, secure_key: 15, load_dates: [date1, date 2, ...]. The filter looks like:

df = spark.read.parquet("adls_storage").where((col(cntr_id) == p[cntr_id]) & (col(load_date).isin(p[load_dates])