Queries with streaming sources must be executed with writeStream.start();

[Bency]

When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error

My aim is to do a lookup for every column in each rows in the streaming data .

steaming_table=spark.readStream.format("delta").table("tableName")

df = (steaming_table.transform(decode_func("a config dic","query to get the lookup value from another table ")))

Where decode_func is the function where I have a custom logic written

[Noopur_Nigam]

Hi @Bency Mathew​ You can use forEachBatch to perform the custom logic on each microbatch. Please refer to below document:

https://docs.databricks.com/structured-streaming/foreach.html#perform-streaming-writes-to-arbitrary-...