topic Re: Incorrect results with df.query() in Data Engineering

Incorrect results with df.query()

plynton — Sat, 08 Oct 2022 15:07:48 GMT

I have tried pulling a single row from a .csv using df.query()

However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:

df = spark.read.option("header",True).csv(data_fldr + "config/CHRGConfig.csv")

df = df.toPandas()

hdrlist = list(df)

xstr = "VERSION == \"" + "STPP" + "\""

print(xstr)

planlist = df.query(xstr)

for zz in planlist:

# I'm looking for non-null values

if not pd.isnull(df.loc[0,zz]):

print(zz)

Re: Incorrect results with df.query()

Hubert-Dudek — Sun, 16 Oct 2022 10:39:42 GMT

Can you include a few rows of your CSV (at least one shouldn't be pulled, and one should)?

Re: Incorrect results with df.query()

plynton — Tue, 18 Oct 2022 13:39:05 GMT

Hubert - I've found a workaround for this, so we can close the discussion.

Thank you!

Re: Incorrect results with df.query()

Anonymous — Thu, 17 Nov 2022 06:06:03 GMT

Hi @Peter Ott

Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

We'd love to hear from you.

Thanks!