Incorrect results with df.query()
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-08-2022 08:07 AM
I have tried pulling a single row from a .csv using df.query()
However, the data being returned doesn't coincide with the data I'm expecting - it is pulling a different row. Here is my code:
df = spark.read.option("header",True).csv(data_fldr + "config/CHRGConfig.csv")
df = df.toPandas()
hdrlist = list(df)
xstr = "VERSION == \"" + "STPP" + "\""
print(xstr)
planlist = df.query(xstr)
for zz in planlist:
#
# I'm looking for non-null values
#
if not pd.isnull(df.loc[0,zz]):
print(zz)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2022 03:39 AM
Can you include a few rows of your CSV (at least one shouldn't be pulled, and one should)?
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2022 06:39 AM
Hubert - I've found a workaround for this, so we can close the discussion.
Thank you!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-16-2022 10:06 PM
Hi @Peter Ott
Does @Hubert Dudek response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?
We'd love to hear from you.
Thanks!