Okay figured the problem. So it works only if I use the .option("kafka.sasl.jaas.config", EH_SASL) Doesn't work when I specify the same details as option("kafka.sasl.username", “myuser”).option("kafka.sasl.password", “mypwd”)
@Kaniz Fatma I am having the same issue.%python
import pyspark.sql.functions as fn
from pyspark.sql.types import StringType
binary_to_string = fn.udf(lambda x: str(int.from_bytes(x, byteorder='big')), StringType())
df = spark.readStream.format("...
