topic Re: Foreign table to delta streaming table in Get Started Discussions

Foreign table to delta streaming table

ksenija — Tue, 23 Jan 2024 08:59:39 GMT

I want to copy a table from a foreign catalog as my streaming table. This is the code I used but I am getting error: Table table_name does not support either micro-batch or continuous scan.;

spark.readStream

.table(table_name)

.writeStream

.trigger(availableNow=True)

.option("checkpointLocation", "dbfs:/folder_path")

.toTable(new_table)

Re: Foreign table to delta streaming table

MLE123 — Tue, 29 Oct 2024 16:04:36 GMT

Bumping this thread because I have the same question and this is still the first result on Google (c. October 2024). Many thanks for anyone who is able to assist!

Re: Foreign table to delta streaming table

sbiales — Wed, 20 Nov 2024 17:31:36 GMT

I also want to bump this! This is my exact problem right now as well.

Re: Foreign table to delta streaming table

cgrant — Tue, 26 Nov 2024 19:45:07 GMT

What is the underlying type of the table you are trying to stream from? Structured Streaming does not currently support streaming reads via JDBC, so reading from MySQL, Postgres, etc are not supported.

If you are trying to perform stream ingestion from such sources, we instead recommend using LakeFlow Connect for supported sources.

Another alternative is to write a Python datasource that performs these streaming reads.

And of course, if your source supports exposing a change log of your data, like the binlog in MySQL or services like AWS DMS, you can set these up and use Databricks Autoloader for efficient incremental ingestion.