How to use SQL Streaming tables

mmenjivar
New Contributor II

‎10-08-2024 09:50 AM

We have been testing the usage of Streaming Tables in our pipelines with different results depending on the streaming source

  • For Streaming Tables reading from read_files everything works as expected
  • For Streaming Tables reading from read_kafka we have some contradictory results when executing in a SQL Warehouse:
    • It works when selecting from read_kafka as in the next block:
SELECT
*
FROM read_kafka(bootstrapServers => 'server',
subscribe => 'topic',
startingOffsets => 'earliest',
`kafka.sasl.mechanism` => 'SCRAM-SHA-512',
`kafka.security.protocol` => 'SASL_PLAINTEXT',
`kafka.sasl.jaas.config` => "kafkashaded.org.apache.kafka.common.security.scram.ScramLoginModule required username='user' password = 'pass';",
failOnDataLoss => 'false'
) limit 10;
    • It doesn't work when I try to create a streaming table using the same query running the script on the same SQL Warehouse:
       
      CREATE OR REFRESH STREAMING TABLE u_marlonmenjivar.test as
      SELECT
      *
      FROM stream read_kafka(bootstrapServers => 'server',
      subscribe => 'topic',
      startingOffsets => 'earliest',
      `kafka.sasl.mechanism` => 'SCRAM-SHA-512',
      `kafka.security.protocol` => 'SASL_PLAINTEXT',
      `kafka.sasl.jaas.config` => "kafkashaded.org.apache.kafka.common.security.scram.ScramLoginModule required username='user' password = 'pass';",
      failOnDataLoss => 'false'
      ) limit 10;
The error that returns is: terminated with exception: kafkashaded.org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeTopics This error is the same for classic, pro and serverless SQL endpoints.

When executed from a notebook it fails with Multipart table names is not supported and when I execute it without schema doesn't fail but it says

To populate your table you must either:
  • Run an existing pipeline using the Delta Live Tables menu
  • Create a new pipeline

According to documentation, the DLT pipeline should be created automatically, any clue on what I'm doing wrong?

0 Kudos