Resolved! In Python, Streaming read by DLT from Hive Table
I am pulling data from Google BigQuery and writing it to a bronze table on an interval. I do this in a separate continuous job because DLT did not like the BigQuery connector calling collect on a dataframe inside of DLT.In Python, I would like to rea...
- 2204 Views
- 1 replies
- 1 kudos
Latest Reply
The below code is a solution. I was missing that I could read from a table with `spark.readStream.format("delta").table("...")`. Simple. Just missed it. This is different than `dlt.read_stream()` which appears in the examples a lot.This is referenced...
- 1 kudos