Queries with streaming sources must be executed with writeStream.start();

Bency — Tue, 02 Aug 2022 14:19:38 GMT

When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error

My aim is to do a lookup for every column in each rows in the streaming data .

steaming_table=spark.readStream.format("delta").table("tableName")

df = (steaming_table.transform(decode_func("a config dic","query to get the lookup value from another table ")))

Where decode_func is the function where I have a custom logic written

Re: Queries with streaming sources must be executed with writeStream.start();

Noopur_Nigam — Fri, 30 Sep 2022 12:44:41 GMT

Hi @Bency Mathew You can use forEachBatch to perform the custom logic on each microbatch. Please refer to below document:

https://docs.databricks.com/structured-streaming/foreach.html#perform-streaming-writes-to-arbitrary-data-sinks-with-structured-streaming-and-foreachbatch

topic Queries with streaming sources must be executed with writeStream.start(); in Data Engineering

Queries with streaming sources must be executed with writeStream.start();

Re: Queries with streaming sources must be executed with writeStream.start();