topic Re: Sample streaming table is failed in Data Engineering

Sample streaming table is failed

shsalami — Thu, 19 Sep 2024 06:24:00 GMT

Running the following databricks sample code in the pipeline:

CREATE OR REFRESH STREAMING TABLE customers

AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv")

I got error:

org.apache.spark.sql.catalyst.ExtendedAnalysisException: Unable to process statement for Table 'customers'.

com.databricks.sql.cloudfiles.errors.CloudFilesIllegalArgumentException: Reading from a Delta table is not supported with this syntax. If you would like to consume data from Delta, please refer to the docs: read a Delta table (https://docs.microsoft.com/azure/databricks/delta/tutorial#read), or read a Delta table as a stream source (https://docs.microsoft.com/azure/databricks/structured-streaming/delta-lake#table-streaming-reads-and-writes). The streaming source from Delta is already optimized for incremental consumption of data.

Re: Sample streaming table is failed

filipniziol — Thu, 19 Sep 2024 07:06:37 GMT

Hi @shsalami ,

The error you're encountering suggests you are attempting to read a Delta table as if it were a collection of CSV files using the cloud_files function.
You may run "DESCRIBE DETAIL customers" and check whether this table exists and it is located in "/databricks-datasets/retail-org/customers/". The other way is to navigate to this location in your cloud storage. It may happen that you already created a Delta table in that location before. The syntax for reading from delta table is different.

Re: Sample streaming table is failed

shsalami — Wed, 25 Sep 2024 05:18:13 GMT

There is no table with that name.

Also, in that folder just the following file exists:

dbfs:/databricks-datasets/retail-org/customers/customers.csv