Databricks

Flow 'user_silver' has FAILED fatally. An error occurred because we detected an update or delete to one or more rows in the source table. Streaming tables may only use append-only streaming sources. If you expect to delete or update rows to the sourc...

I'm envisioning a live streaming pipeline.The bronze, or data ingestion, is being fetched using the directory listing mode of the autoloader.I'm not using File Notification Mode because I detect about 2-300 data changes per hour.I'm thinking about im...

I have already checked the documentation below The documentation below does not describe how to write to streaming.Is there a way to write the gold table (type is streaming table), which is the output of the streaming pipeline of Delta Live Tables in...

Do you know why the userIdentity is anonymous in AWS Cloudtail's logs even though I have specified an instance profile?

  from pyspark.sql import functions as F from pyspark.sql import types as T from pyspark.sql import DataFrame, Column from pyspark.sql.types import Row import dlt S3_PATH = 's3://datalake-lab/xxxx/' S3_SCHEMA = 's3://datalake-lab/xxxx/schemas/' @dl...