Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-23-2025 09:51 AM
I'm having difficulty adding a mask function to columns while creating streaming tables using the DLT Python method create_streaming_table() like this but it does not work, the streaming table is created but no column is masked:
def prepare_column_properties_struct(table_contract: dict) -> StructType:
struct_fields = []
for column_properties in table_contract["models"]["columns"]:
column_name = column_properties["name"]
column_type = column_properties["type"]
column_nullable = not column_properties["required"]
column_comment = column_properties["comment"]
column_mask = column_properties["mask"]
struct_fields.append(
StructField(
name=column_name,
dataType=parse_data_type(column_type),
nullable=column_nullable,
metadata={"comment": column_comment, "mask": "mask_all"},
)
)
return StructType(struct_fields)
dlt.create_streaming_table(
name="account",
schema=prepare_column_properties_struct(data_contract),
)How do I go about this? Maybe I'm not using the correct metadata key in the StructField? The doc is not helping.