cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

create flow for streaming table

bigger_dave
New Contributor II

Hi Team.
I'm following the example code to create flows, which is here.

When I create the streaming table without a query (see code below):

CREATE OR REFRESH STREAMING TABLE target_table;

I get the error "The operation CREATE WITHOUT A QUERY is not allowed: The operation is not supported on Streaming Tables.".

Does anyone know what I am doing wrong?



 

1 REPLY 1

BigRoux
Databricks Employee
Databricks Employee
The error "The operation CREATE WITHOUT A QUERY is not allowed: The operation is not supported on Streaming Tables" occurs because the CREATE OR REFRESH STREAMING TABLE statement requires a query to define the data source for the streaming table. Streaming tables necessitate a query that specifies how their data is populated, rather than being created without a query specification.
Here are the key points related to this issue:
  1. Streaming tables are designed for handling incremental or streaming data processing. The CREATE STREAMING TABLE command must include an AS query clause that provides a streaming query to process and populate the table. Omitting the query is not supported.
  2. The CREATE OR REFRESH STREAMING TABLE statement is valid for refreshing existing streaming tables with new data from the defined query, but still requires the query to be specified during creation.
  3. For initialization, examples of valid syntax include: sql CREATE OR REFRESH STREAMING TABLE example_table AS SELECT * FROM STREAM read_files('<path_to_file>', '<options>'); This type of statement specifies the data source and streaming query configuration.
  4. If you simply want to refresh an existing table, use the REFRESH STREAMING TABLE <table_name> command or explicitly define it with a query for a full refresh.
You need to revise your statement to include a streaming query when creating the table. Let me know if you'd like help constructing it!
 
DLT uses a simple declaritive language when using SQL. In short, it is essentially a CTAS statement like you would use in SQL.   Check out our documenation.
 
Hope this helps.  Lou.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now