Databricks streaming dataframe into Snowflake

Anonymous · ‎07-01-2023

Any suggestions on how to stream data from databricks into snowflake?. Is snowpipe is the only option?. Snowpipe is not faster since it runs copy into in a small batch intervals and not in few seconds. If no option other than snowpipe, how to call it from databricks notebook?. Thx in advance for your help.

Below did not work since my dataDF was streaming dataframe.

import net.snowflake.spark.snowflake.Utils

dataDF.write

.format("snowflake")

.options(sfOptions)

.option("dbtable", "demokinesisstream")

.mode(SaveMode.Append)

.save()