<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Missing spark structured streaming metadata blobs? in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176531#M57169</link>
    <description>&lt;P&gt;I don't have a heavy dependency on the databricks jobs platform yet.&amp;nbsp; Our spark jobs have previously been hosted in many places (synapse, HDI, fabric, and here).&amp;nbsp; It seems dangerous to use the databricks-jobs component for such a crucial detail, that ties so closely to the internal workings of a spark job.&amp;nbsp; I'd rather that concurrency be managed within the spark job itself.&lt;/P&gt;</description>
    <pubDate>Mon, 16 Nov 2026 14:27:16 GMT</pubDate>
    <dc:creator>DB1To3</dc:creator>
    <dc:date>2026-11-16T14:27:16Z</dc:date>
    <item>
      <title>Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176124#M57094</link>
      <description>&lt;P&gt;I'm exploring spark structured streaming, and contemplating uses for it.&lt;/P&gt;&lt;P&gt;I'd like to launch a job on a recurring basis from an external scheduler.&amp;nbsp; It will launch a spark job with trigger (availableNow=True).&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;I'm concerned about overlapping executions, and corruption of the "&lt;STRONG&gt;checkpointLocation&lt;/STRONG&gt;".&amp;nbsp; As I was reading the docs, it seems like a folder is needed for that:&lt;STRONG&gt;_spark_metadata&lt;/STRONG&gt;.&amp;nbsp; It has benefits like:&lt;/P&gt;&lt;UL&gt;&lt;LI&gt;Protect against concurrent processing of the same query to the same target&lt;/LI&gt;&lt;LI&gt;Ensure "exactly once" processing of inputs&lt;/LI&gt;&lt;/UL&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;However whenever I search for that folder (_spark_metadata) in blob storage I'm unable to find it.&amp;nbsp; I suspect this has something to do with "toTable" that delivers data into a UC managed-table.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="DB1To3_0-1794341533309.png" style="width: 400px;"&gt;&lt;img src="https://community.databricks.com/t5/image/serverpage/image-id/33237i70221A90280B8C15/image-size/medium?v=v2&amp;amp;px=400" role="button" title="DB1To3_0-1794341533309.png" alt="DB1To3_0-1794341533309.png" /&gt;&lt;/span&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;Is there any way I can deliver data to the UC managed-table &lt;STRONG&gt;without losing the benefits of the&amp;nbsp;_spark_metadata&lt;/STRONG&gt; folder?&amp;nbsp; I'd like to have a consistent set of code that can be written to deliver data to a flexible target, without losing the _spark_metadata folder in the process.&amp;nbsp; I don't see why the delivery of streaming to a table is mutually excluding the ability to generate the _spark_metadata blobs.&amp;nbsp; Any tips would be appreciated.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;BR /&gt;&lt;SPAN&gt;&lt;BR /&gt;&amp;nbsp;&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Tue, 10 Nov 2026 20:17:27 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176124#M57094</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-10T20:17:27Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176144#M57097</link>
      <description>&lt;P&gt;&lt;SPAN&gt;Hi &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/3844"&gt;@DB1To3&lt;/a&gt;,&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;You do not lose anything here. The &lt;/SPAN&gt;&lt;SPAN&gt;_spark_metadata&lt;/SPAN&gt;&lt;SPAN&gt; folder belongs to the Spark file sink (for example &lt;/SPAN&gt;&lt;SPAN&gt;format("parquet").start(path)&lt;/SPAN&gt;&lt;SPAN&gt;), so it does not appear when you write with &lt;/SPAN&gt;&lt;SPAN&gt;toTable&lt;/SPAN&gt;&lt;SPAN&gt; to a Delta table. For a Delta sink, the Delta transaction log does that job. The Databricks docs say it "guarantees exactly-once processing, even when there are other streams or batch queries running concurrently against the table." Your &lt;/SPAN&gt;&lt;SPAN&gt;checkpointLocation&lt;/SPAN&gt;&lt;SPAN&gt; still keeps the offsets and commits of each micro-batch, so the next &lt;/SPAN&gt;&lt;SPAN&gt;availableNow&lt;/SPAN&gt;&lt;SPAN&gt; run continues where the last run stopped. If one of your flexible targets is a plain file path, Spark creates &lt;/SPAN&gt;&lt;SPAN&gt;_spark_metadata&lt;/SPAN&gt;&lt;SPAN&gt; there by itself, so the same code works for both target types. The docs also say that each query must have its own checkpoint, and that queries must never share one. So the real risk is two runs of the same query at the same time, not the missing folder. A simple guard: put the code in a Lakeflow Job and let your external scheduler call the Jobs API "run now". New jobs have "maximum concurrent runs" set to 1, so an overlapping trigger is queued or skipped and never runs in parallel.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Docs: &lt;A href="https://docs.databricks.com/aws/en/structured-streaming/delta-lake" target="_blank"&gt;https://docs.databricks.com/aws/en/structured-streaming/delta-lake&lt;/A&gt;&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 05:48:39 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176144#M57097</guid>
      <dc:creator>Yougasundar</dc:creator>
      <dc:date>2026-11-11T05:48:39Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176158#M57098</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/3844"&gt;@DB1To3&lt;/a&gt;&amp;nbsp;Just make this concurrent run to 1 for your streaming jobs are your problem is sorted , it wont corrupt checkpoint locatiion&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="saurabh18cs_0-1794390542784.png" style="width: 400px;"&gt;&lt;img src="https://community.databricks.com/t5/image/serverpage/image-id/33244iA9D64F56B78B96C1/image-size/medium?v=v2&amp;amp;px=400" role="button" title="saurabh18cs_0-1794390542784.png" alt="saurabh18cs_0-1794390542784.png" /&gt;&lt;/span&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 09:49:19 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176158#M57098</guid>
      <dc:creator>saurabh18cs</dc:creator>
      <dc:date>2026-11-11T09:49:19Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176171#M57099</link>
      <description>&lt;P&gt;Thanks for the thorough answer.&lt;BR /&gt;When we use file/blob output, we use&amp;nbsp; &lt;STRONG&gt;.option("path", xyz)&lt;/STRONG&gt; and that is the place where the _spark_metadata lives.&amp;nbsp; If we stream to a table then we don't use that option anymore - and we lose the folder. My concern was that when we switch to tables we lose something we otherwise had when streaming to files.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&amp;gt;&amp;gt; So the real risk is two runs of the same query at the same time, not the missing folder.&amp;nbsp;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;Exactly.&amp;nbsp; Much of my understanding is from LLM responses so take with a grain of salt, but I heard the &lt;STRONG&gt;_spark_metadata&lt;/STRONG&gt; manages concurrency and allows only one writeStream to be delivering data to that query-and-sink at any time.&amp;nbsp; It doesn't seem to be very well documented, possibly because it is a feature of the sink instead of a feature of spark structured streaming as a whole.&lt;/P&gt;&lt;P&gt;It continues to baffle me that there is no built-in concurrency control for something so fundamental as this (preventing the writing of the same query to the same sink by concurrent drivers).&amp;nbsp; I suppose the Lakeflow Job approach works too, but it seems a bit overkill.&amp;nbsp; A blob-lease seems better, but I haven't found the discussions about doing it that way (yet).&amp;nbsp; Possibly because that is what &lt;STRONG&gt;_spark_metadata is&lt;/STRONG&gt; doing internally?&amp;nbsp; I'm a bit new to databricks; does everyone use lakeflow jobs for critical sections?&amp;nbsp; I appreciate the creative approach, but it does not seem like a versatile way to solve the generalized need for one thing happening at a time in this platform.&amp;nbsp; Maybe everyone will start using lakebase tables for that?&amp;nbsp; &amp;nbsp;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 12:04:12 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176171#M57099</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-11T12:04:12Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176172#M57100</link>
      <description>&lt;P&gt;What if I'm not using lakeflow jobs?&amp;nbsp; Eg. what if I'm using the REST api to submit jobs (POST /api/2.2/jobs/runs/submit to create ephemeral runs)&lt;/P&gt;&lt;P&gt;Are you also confirming that spark structured streaming won't protect me from duplicate/overlapping executions of the same query-and-sink?&amp;nbsp; It is odd that I don't find more discussions about this.&amp;nbsp; It is hard to google, especially prior to experiencing the symptoms of the concurrency conflicts (and potential corruption).&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 12:10:46 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176172#M57100</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-11T12:10:46Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176177#M57102</link>
      <description>&lt;P&gt;&lt;SPAN&gt;Hi &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/3844"&gt;@DB1To3&lt;/a&gt;,&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Good follow-up, and you are right that you lose nothing important. The _spark_metadata folder is the file sink's own commit log: it records which files each batch committed, so that readers of that path skip files from failed or repeated batches. It is not a lock. With a table sink, the Delta transaction log does that commit job, which is why the folder is not needed there. The concurrency guard you are looking for lives in the checkpoint, and it works the same for file and table sinks. The Databricks error reference lists two errors for this case: CONCURRENT_QUERY ("Another instance of this query ... was just started by a concurrent session") and CONCURRENT_STREAM_LOG_UPDATE ("Concurrent update to the log. Multiple streaming jobs detected for &amp;lt;batchId&amp;gt;. Please make sure only one streaming job runs on a specific checkpoint location at a time."). So if two drivers start the same query on the same checkpoint, the second one is expected to fail with one of these errors instead of silently writing the batch again. This is fail-fast detection, not a queue, so your scheduler should either avoid the overlap or treat that error as "already running". That is why the Lakeflow Job setting is common for scheduled streams: with maximum concurrent runs set to 1, an overlapping trigger is queued or skipped before a second driver starts, and your external scheduler can still start it with the Jobs API "run now". A blob lease or a lock row in a Lakebase table (it is Postgres, so normal row locks work) is fine for a general "one at a time" need, but for streaming the checkpoint already gives you the protection that the _spark_metadata folder never did.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Docs: &lt;A href="https://docs.databricks.com/aws/en/error-messages/error-classes" target="_blank"&gt;https://docs.databricks.com/aws/en/error-messages/error-classes&lt;/A&gt;&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 13:07:41 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176177#M57102</guid>
      <dc:creator>Yougasundar</dc:creator>
      <dc:date>2026-11-11T13:07:41Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176178#M57103</link>
      <description>&lt;P&gt;Thanks.&amp;nbsp; I will do some testing and see if I can get those concurrency error messages to appear.&amp;nbsp; It is one of the times when you really, really want to see those errors, because things would be a lot worse without them!&lt;BR /&gt;&lt;BR /&gt;If you know of any blogs or docs that I'm overlooking, please let me know.&amp;nbsp; Your simple statement, "&lt;SPAN&gt;concurrency guard you are looking for lives in the checkpoint", is not something I found yet in my reading.&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Wed, 11 Nov 2026 13:20:57 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176178#M57103</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-11T13:20:57Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176426#M57142</link>
      <description>&lt;P&gt;&lt;SPAN&gt;Hi &lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/3844"&gt;@DB1To3&lt;/a&gt;&lt;/SPAN&gt;&lt;SPAN&gt;,&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Thanks for accepting the answer. On the docs you asked for: the Structured Streaming checkpoints page says "Each query must have a different checkpoint location. Multiple queries should never share the same location", and the two error messages are in the Databricks error reference (search for CONCURRENT_QUERY and CONCURRENT_STREAM_LOG_UPDATE). I have not seen a page that explains how they are detected, so your test is a good idea: start the same availableNow query twice, on two compute resources, with the same checkpoint, and you should see one of the two errors. On your runs/submit question: the API reference says that runs submitted with that endpoint "are not saved as a job", so a job's "maximum concurrent runs" setting cannot protect them. The endpoint has an optional idempotency_token, and the same page says "Databricks guarantees that exactly one run is launched with that idempotency token". If your scheduler builds the token from the schedule slot (for example the query name plus the planned start time), a retry or a double submit for the same slot cannot start a second run. It does not stop the next slot from starting while the previous run is still going, so for that the checkpoint errors remain your guard, or you can use jobs/create plus jobs/run-now, which the same page suggests.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Docs: &lt;A href="https://docs.databricks.com/api/workspace/jobs/submit" target="_blank"&gt;https://docs.databricks.com/api/workspace/jobs/submit&lt;/A&gt;&lt;/SPAN&gt;&lt;/P&gt;</description>
      <pubDate>Sun, 15 Nov 2026 04:34:37 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176426#M57142</guid>
      <dc:creator>Yougasundar</dc:creator>
      <dc:date>2026-11-15T04:34:37Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176519#M57165</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/3844"&gt;@DB1To3&lt;/a&gt;&amp;nbsp;I think better approach would be to deploy your job as part of prod release deployment management process and handle job run via&amp;nbsp;POST /api/2.2/jobs/run-now&lt;/P&gt;&lt;P&gt;this way you can control max concurrent runs or you have to add a logic to validate if existing run is there or not&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 13:28:44 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176519#M57165</guid>
      <dc:creator>saurabh18cs</dc:creator>
      <dc:date>2026-11-16T13:28:44Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176530#M57168</link>
      <description>&lt;P&gt;Thanks again for your help.&lt;/P&gt;&lt;P&gt;&lt;STRONG&gt;&amp;gt;&amp;gt;&amp;nbsp;On your runs/submit question: the API reference says that runs submitted with that endpoint "are not saved as a job", so a job's "maximum concurrent runs" setting cannot protect them.&amp;nbsp;&lt;/STRONG&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;I will probably use a simpler approach like a blob lease or OTLP pessimistic record lock.&amp;nbsp; These are EXTREMELY common ways of synchronization and don't require a dependency on the databricks jobs platform (ie.&amp;nbsp; we may one day use airflow or some other external tool to launch our streams.&amp;nbsp; And if/when that happens we wouldn't want to revisit any concurrency-control requirements.)&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 14:24:11 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176530#M57168</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-16T14:24:11Z</dc:date>
    </item>
    <item>
      <title>Re: Missing spark structured streaming metadata blobs?</title>
      <link>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176531#M57169</link>
      <description>&lt;P&gt;I don't have a heavy dependency on the databricks jobs platform yet.&amp;nbsp; Our spark jobs have previously been hosted in many places (synapse, HDI, fabric, and here).&amp;nbsp; It seems dangerous to use the databricks-jobs component for such a crucial detail, that ties so closely to the internal workings of a spark job.&amp;nbsp; I'd rather that concurrency be managed within the spark job itself.&lt;/P&gt;</description>
      <pubDate>Mon, 16 Nov 2026 14:27:16 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/missing-spark-structured-streaming-metadata-blobs/m-p/176531#M57169</guid>
      <dc:creator>DB1To3</dc:creator>
      <dc:date>2026-11-16T14:27:16Z</dc:date>
    </item>
  </channel>
</rss>

