topic Re: Delta live table UC Kinesis: options overwriteschema, ignorechanges not supported for data sourc in Data Engineering

Delta live table UC Kinesis: options overwriteschema, ignorechanges not supported for data sourc

alemo — Tue, 04 Jul 2023 06:25:59 GMT

I try to build a DLT in UC with Kinesis as producer.

My first table looks like:

@dlt.create_table(
 table_properties={
     "pipelines.autoOptimize.managed": "true"
 },
 spark_conf={"spark.databricks.delta.schema.autoMerge.enabled": "true"},
)

def feed_chunks():
return (
 spark.readStream.format("kinesis")
 .option("streamName", stream_name)
 .option("ignoreChanges", "true")
 .option("overwriteSchema", "true")
 .option("roleArn", "arn:aws:iam::xxxxx:role/xxxxx-read-role")
 .option("region", "eu-west-1")
 .load()
 )

And I get this error:

pyspark.errors.exceptions.AnalysisException: [UNSUPPORTED_STREAMING_OPTIONS_PERMISSION_ENFORCED] 
Streaming options overwriteschema, ignorechanges are not supported for 
data source kinesis on a shared cluster.

Is this a know limitation of DLT x UC ?

The same DLT with simple hive metastore works fine.

Re: Delta live table UC Kinesis: options overwriteschema, ignorechanges not supported for data sourc

sawya — Sun, 30 Jul 2023 20:08:28 GMT

Hi,

Did you found a solution ? I have the same problem

Thanks !

Re: Delta live table UC Kinesis: options overwriteschema, ignorechanges not supported for data sourc

Corbin — Wed, 23 Aug 2023 13:43:56 GMT

If you use the "Preview" Channel in the "Advanced" section of the DLT Pipeline, this error should resolve itself. This fix is planned to make it into the "Current" channel by Aug 31, 2023

Re: Delta live table UC Kinesis: options overwriteschema, ignorechanges not supported for data sourc

sawya — Mon, 04 Sep 2023 12:01:34 GMT

Hello !

My pipeline is already in Advanced mode because I'm using Unity Catalog in storage options