<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:taxo="http://purl.org/rss/1.0/modules/taxonomy/" version="2.0">
  <channel>
    <title>topic Re: Databricks as a &amp;quot;pure&amp;quot; data streaming software like Confluent in Data Engineering</title>
    <link>https://community.databricks.com/t5/data-engineering/databricks-as-a-quot-pure-quot-data-streaming-software-like/m-p/93448#M38707</link>
    <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/124839"&gt;@noorbasha534&lt;/a&gt;&amp;nbsp;,&lt;/P&gt;&lt;P&gt;It depends on what you're asking for.&amp;nbsp;&lt;SPAN&gt;Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in message distribution.&amp;nbsp;&lt;BR /&gt;But if you're comparing Kafka Streams&amp;nbsp;&lt;SPAN&gt;(which is Kafka offering for stream processing) &lt;/SPAN&gt;with Apache Spark Structured Streaming (which Databricks uses for stream processing)&amp;nbsp; then yes, I think Databricks streaming capablilities are top-notch and you can use it instead of Kafka and you'll be happy.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;As for Apache Flink, it is known for low-latency, stateful, and complex event processing. If your streaming use case involves complex operations, maybe Flink would be a better choice. But with the intensive development of spark strucutred streaming, this boundary is blurring.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
    <pubDate>Thu, 10 Oct 2024 14:29:03 GMT</pubDate>
    <dc:creator>szymon_dybczak</dc:creator>
    <dc:date>2024-10-10T14:29:03Z</dc:date>
    <item>
      <title>Databricks as a "pure" data streaming software like Confluent</title>
      <link>https://community.databricks.com/t5/data-engineering/databricks-as-a-quot-pure-quot-data-streaming-software-like/m-p/93422#M38703</link>
      <description>&lt;P&gt;Dears&lt;/P&gt;&lt;P&gt;I was wondering if anyone has leveraged Databricks as a "pure" data streaming software in place of Confluent, Flink, Kafka etc.&lt;/P&gt;&lt;P&gt;I see the reference architectures placing Databricks on the data processing side mostly once data is made available by Confluent or Flink or Kafka.&lt;/P&gt;&lt;P&gt;Appreciate if you can share your insights.&lt;/P&gt;</description>
      <pubDate>Thu, 10 Oct 2024 12:13:13 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/databricks-as-a-quot-pure-quot-data-streaming-software-like/m-p/93422#M38703</guid>
      <dc:creator>noorbasha534</dc:creator>
      <dc:date>2024-10-10T12:13:13Z</dc:date>
    </item>
    <item>
      <title>Re: Databricks as a "pure" data streaming software like Confluent</title>
      <link>https://community.databricks.com/t5/data-engineering/databricks-as-a-quot-pure-quot-data-streaming-software-like/m-p/93448#M38707</link>
      <description>&lt;P&gt;Hi&amp;nbsp;&lt;a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/124839"&gt;@noorbasha534&lt;/a&gt;&amp;nbsp;,&lt;/P&gt;&lt;P&gt;It depends on what you're asking for.&amp;nbsp;&lt;SPAN&gt;Kafka is primarily a messaging system, optimized for handling high-throughput, distributed message logs. Databricks can read from Kafka as a data source but doesn't replace Kafka's role in message distribution.&amp;nbsp;&lt;BR /&gt;But if you're comparing Kafka Streams&amp;nbsp;&lt;SPAN&gt;(which is Kafka offering for stream processing) &lt;/SPAN&gt;with Apache Spark Structured Streaming (which Databricks uses for stream processing)&amp;nbsp; then yes, I think Databricks streaming capablilities are top-notch and you can use it instead of Kafka and you'll be happy.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&amp;nbsp;As for Apache Flink, it is known for low-latency, stateful, and complex event processing. If your streaming use case involves complex operations, maybe Flink would be a better choice. But with the intensive development of spark strucutred streaming, this boundary is blurring.&lt;/P&gt;&lt;P&gt;&amp;nbsp;&lt;/P&gt;</description>
      <pubDate>Thu, 10 Oct 2024 14:29:03 GMT</pubDate>
      <guid>https://community.databricks.com/t5/data-engineering/databricks-as-a-quot-pure-quot-data-streaming-software-like/m-p/93448#M38707</guid>
      <dc:creator>szymon_dybczak</dc:creator>
      <dc:date>2024-10-10T14:29:03Z</dc:date>
    </item>
  </channel>
</rss>

