Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2022 01:29 PM
I predefined my schema for a Delta Live Table Autoload. This included comments for some attributes. When performing a standard readStream, my comments appear, but when in Delta Live Tables I get no comments. Is there anything I need to do get comments to appear?
Schema definition:
schema = StructType([
StructField("uuid",StringType(),True, {'comment': "Unique customer id"}),
StructField("GPS",StringType(),True)])Delta Live Table Stream:
@dlt.table(name="test_bronze",
comment = "test account data incrementally ingested from S3 Raw landing zone",
table_properties={
"quality": "bronze"
}
)
# Stream data
#@dlt.table
def test_bronze():
return (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv)
.option("header", "True")
.schema(schema)
.load(data_source)
)But no comments in data:
Labels:
- Labels:
-
Delta Live Tables