cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks + Snowflake Snowpipe Streaming

ptutak
New Contributor III

Does anyone know whether it is possible to use Databricks Snowflake Connector together with the latest Snowflake feature which is Snowpipe Streaming?

5 REPLIES 5

artsheiko
Valued Contributor II
Valued Contributor II

At its core, Snowpipe is a tool to copy data into Snowflake from cloud storage. Snowpipe is not about a streaming, but about how to batch load data from cloud storage into a table on a recurring basis.

Databricks has a similar feature that we call Auto Loader. Auto loader enables developers to create a Spark Structured Streaming pipeline with cloud files as a data source.

Use Structured Streaming or Delta Live Tables to implement streaming use-cases.

ptutak
New Contributor III

The use case that we're trying to solve is loading data from Kafka into Snowflake using Databricks, we could use the Databricks Snowflake Connector but only if it's able to use Snowpipe Streaming underneath.

artsheiko
Valued Contributor II
Valued Contributor II

why do you need Snowpipe at all ? Connect to Kafka directly from Databricks : check https://www.dbdemos.ai/demo.html?demoName=streaming-sessionization

ptutak
New Contributor III

With typical Snowflake Connector we cannot omit the Snowflake Warehouse which adds to the cost of the whole operation, whereas when we're using Snowpipe Streaming we can omit the Warehouse and reduce the cost: snowpipe-streaming-cost .

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.