I'm exploring spark structured streaming, and contemplating uses for it.
I'd like to launch a job on a recurring basis from an external scheduler. It will launch a spark job with trigger (availableNow=True).
I'm concerned about overlapping executions, and corruption of the "checkpointLocation". As I was reading the docs, it seems like a folder is needed for that:_spark_metadata. It has benefits like:
- Protect against concurrent processing of the same query to the same target
- Ensure "exactly once" processing of inputs
However whenever I search for that folder (_spark_metadata) in blob storage I'm unable to find it. I suspect this has something to do with "toTable" that delivers data into a UC managed-table.
Is there any way I can deliver data to the UC managed-table without losing the benefits of the _spark_metadata folder? I'd like to have a consistent set of code that can be written to deliver data to a flexible target, without losing the _spark_metadata folder in the process. I don't see why the delivery of streaming to a table is mutually excluding the ability to generate the _spark_metadata blobs. Any tips would be appreciated.
