I try to build a DLT in UC with Kinesis as producer.
My first table looks like:
@dlt.create_table(
table_properties={
"pipelines.autoOptimize.managed": "true"
},
spark_conf={"spark.databricks.delta.schema.autoMerge.enabled": "true"},
)
def feed_chunks():
return (
spark.readStream.format("kinesis")
.option("streamName", stream_name)
.option("ignoreChanges", "true")
.option("overwriteSchema", "true")
.option("roleArn", "arn:aws:iam::xxxxx:role/xxxxx-read-role")
.option("region", "eu-west-1")
.load()
)
And I get this error:
pyspark.errors.exceptions.AnalysisException: [UNSUPPORTED_STREAMING_OPTIONS_PERMISSION_ENFORCED]
Streaming options overwriteschema, ignorechanges are not supported for
data source kinesis on a shared cluster.
Is this a know limitation of DLT x UC ?
The same DLT with simple hive metastore works fine.
