cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Databricks + Snowflake Snowpipe Streaming

ptutak
New Contributor III

Does anyone know whether it is possible to use Databricks Snowflake Connector together with the latest Snowflake feature which is Snowpipe Streaming?

5 REPLIES 5

artsheiko
Databricks Employee
Databricks Employee

At its core, Snowpipe is a tool to copy data into Snowflake from cloud storage. Snowpipe is not about a streaming, but about how to batch load data from cloud storage into a table on a recurring basis.

Databricks has a similar feature that we call Auto Loader. Auto loader enables developers to create a Spark Structured Streaming pipeline with cloud files as a data source.

Use Structured Streaming or Delta Live Tables to implement streaming use-cases.

ptutak
New Contributor III

The use case that we're trying to solve is loading data from Kafka into Snowflake using Databricks, we could use the Databricks Snowflake Connector but only if it's able to use Snowpipe Streaming underneath.

artsheiko
Databricks Employee
Databricks Employee

why do you need Snowpipe at all ? Connect to Kafka directly from Databricks : check https://www.dbdemos.ai/demo.html?demoName=streaming-sessionization

ptutak
New Contributor III

With typical Snowflake Connector we cannot omit the Snowflake Warehouse which adds to the cost of the whole operation, whereas when we're using Snowpipe Streaming we can omit the Warehouse and reduce the cost: snowpipe-streaming-cost .

artsheiko
Databricks Employee
Databricks Employee

@Piotr Tutakโ€‹ , I believe you don't need Snowflake at all - just use source files / events from your Data Lake / message brocker to process it whithin Databricks. Auto-loader that might be combined with DLT.

How to + demo.

Auto loader doc

DLT doc

With Auto loader you'll benefit also from the possibility to work in streaming mode and switch to batch if needed. DLT allows to guarantee data quality. Obviously much more value added features might be listed here if we compare vs snowpipe. Among them you'll be able to estimate and control costs - something that is still not possible with snowpipe, unfortunately.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group