@animeshjain I think the error is coming from the local Python bundle parser, not from the schema itself, your CLI is showing a schema where lifecycle is valid for volume, but may be the installed databricks.bundles.volumes runtime that actually pars...
Hey @alejandro_jaram, You can check if the DLT/Lakeflow event log for the affected materialized view. Compare one fast successful run with one slow/stuck run and check the planning_information details. Specifically, look for whether the MV is using a...
I think this is more like an incremental refresh issue than a generic serverless memory issue.Since the pipeline completes in around 20 minutes with a full refresh, but the normal weekly run runs for hours and then fails with OOM, I would first recom...
Hey @prasuanu1222, For 10,000 daily users and 100+ requests per minute, plain iframe embedding is probably not enough by itself.Databricks has workspace level throughput limits for Genie. For iframe/UI access, the documented limit is around 20 questi...
Yes, we can build a continuous streaming pipeline using open source Spark. The main thing is to use Spark Structured Streaming, not a normal batch read. For Kafka streaming, we need to use spark.readStream, then write using writeStream, and keep the ...
