When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error
My aim is to do a lookup for every column in each rows in the streaming data .
steaming_table=spark.readStream.format("delta").table("tableName")
df = (steaming_table.transform(decode_func("a config dic","query to get the lookup value from another table ")))
Where decode_func is the function where I have a custom logic written