DLT Notebook Error - Queries with streaming sources must be executed with writeStream.start();

[JJ_LVS1]

I'm trying to parse incoming stream files in DLT which have variable length records. I'm getting the error:

Queries with streaming sources must be executed with writeStream.start();

Notebook code

@dlt.table (
    comment="xAudit Parsed"
)
def b_table_parsed():
    df = dlt.readStream("dlt_able_raw_view")
          
    for i in range(df.select(F.max(F.size('split_col'))).collect()[0][0]):
        df = df.withColumn("col"+str(i),df["split_col"][i])
    
    df = (df
          .drop("value","split_col")
         )
      
    return df

This all works fine against the actual source text files or a delta table using the interactive cluster but when I put it in DLT and and the source is streaming files from autoloader, it doesn't like it. I assume it's stream related.

I saw a different post about using foreach maybe but that was using writeStream and not sure if I can use it to return in a DLT table. I'm very new to python, streaming and DLT so would appreciate if anyone has a detailed solution.

[Murthy1 · Accepted solution]

Hello @Jason Johnson​ ,

It seems like, you want to have some transformations on the incoming micro batches - in that case I guess, you cannot achieve it through Delta live tables. You have to use a foreachbatch and then use writestream to achieve this.

[Anonymous]

Hi @Jason Johnson​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!