don't see why it would generate an error as it works fine.

Perhaps, as a test, serialize the dataframe (write it and read again) and then try the filter + count.

If that works, it is probably your query plan which might be too complex, or the AQE which goes crazy.