- 1861 Views
- 4 replies
- 0 kudos
Resolved! Materialized view creation fails
Hi,I have ran into a problem when creating materialized view.Here's my simple query I'm trying to run:%sql create or replace materialized view catalog.schema.mView_test as select * from catalog.schema.table limit 10;I'm getting following error:Encoun...
- 1861 Views
- 4 replies
- 0 kudos
- 0 kudos
There are multiple requirements for materialized views. You can check belowYou must use a Unity Catalog enabled pro or serverless SQL warehouse.To incrementally refresh a materialized view from Delta tables, the source tables must have row tracking e...
- 0 kudos
- 2259 Views
- 3 replies
- 1 kudos
How to implement MERGE operations in Lakeflow Declarative Pipelines
Hey everyone,We’ve been using Autoloader extensively for a while, and now we’re looking to transition to full Lakeflow Declarative Pipelines. From what I’ve researched, the reader part seems straightforward and clear.For the writer, I understand that...
- 2259 Views
- 3 replies
- 1 kudos
- 1 kudos
Use APPLY CHANGES INTO (SQL) or dlt.apply_changes() (Python). This is the declarative replacement for foreachBatch MERGE logic in pipelines import dlt from pyspark.sql.functions import col @dlt.table(name="bronze_events") def bronze_events(): re...
- 1 kudos
- 849 Views
- 2 replies
- 1 kudos
Resolved! pipeline config DAB
I am deploying DLT pipeline in dev environment using DABs. source code is in a python script file. In the pipeline's yml file the configuration key is set to true(with all correct indentations), yet the pipeline isnt deploying in the continuous mode....
- 849 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @prakharsachan ,Continuous must be set inside the pipeline resource definition, not under configuration.The configuration block in a SDP (former DLT) pipeline definition is for Spark/pipeline settings (key-value string pairs passed to the runtime)...
- 1 kudos
- 1210 Views
- 2 replies
- 0 kudos
Cannot create streaming table
Hi,while trying to create a streaming table in sql notebook, i am getting below error. please can you assist to fix it.The operation CREATE is not allowed: Cannot CREATE the Streaming Table `my_catalog`.`test_schema`.`emp` in Serverless Generic Compu...
- 1210 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @databrick_enthu , The error is unfortunately misleading. The Materialized View / Streaming Table on Serverless Generic Compute is not yet available for a Preview. For now, you have two possibilities: Attach the notebook to a Serverless SQL Wareh...
- 0 kudos
- 1274 Views
- 2 replies
- 0 kudos
Resolved! Bug Report: Incorrect “Next Run Time” Calculation for Interval Periodic Schedules
SummaryWhen configuring a Jobs schedule → Interval periodic trigger in Databricks, the “Next run” timestamp changes inconsistently when the interval value is modified, even though the base schedule (start time) remains the same. The next run appears ...
- 1274 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @Ashwin_DSA for that detailed answer, that surely solves my query but I see this as a classic UX bug disguised as feature
- 0 kudos
- 683 Views
- 1 replies
- 1 kudos
Resolved! Missing workspaces in workspaces_latest but present in audit
While validating workspace coverage, we observed that some workspace_id exist in system.access.audit but return NULL in system.access.workspaces_latest. select distinct a.workspace_id, w.workspace_name from system.access.audit a left join system.acc...
- 683 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Danish11052000, Yes... This is expected and documented behaviour, not a bug. system.access.workspaces_latest contains only active workspaces in the account. When a workspace is cancelled/removed from the account, its row is removed from this tab...
- 1 kudos
- 2630 Views
- 11 replies
- 2 kudos
Resolved! What has happened to syntax coloring for SQL queries???
What has happened to syntax coloring for SQL queries??? It seems that everything is in color blue which is confusing and hard to read the code...
- 2630 Views
- 11 replies
- 2 kudos
- 1525 Views
- 2 replies
- 2 kudos
Resolved! Redshift to Databricks Migration with Lakebridge
We are currently performing an assessment for a client’s Redshift to Databricks migration, and we would like to better understand the enhanced capabilities of Lakebridge for this use case.We would appreciate clarification on the following points:Scop...
- 1525 Views
- 2 replies
- 2 kudos
- 2 kudos
There is a nice course on Partner Academy as well . It uses SQL Server as a target system for migration but you can follow the same steps for Redshift as well . https://partner-academy.databricks.com/learn/courses/4326/lakebridge-for-sql-source-syste...
- 2 kudos
- 3236 Views
- 5 replies
- 1 kudos
Resolved! Registering Delta tables from external storage GCS , S3 , Azure Blob in Databricks Unity Catalog
Hi everyone,I am currently working on a migration project from Azure Databricks to GCP Databricks, and I need some guidance from the community on best practices around registering external Delta tables into Unity Catalog.Currenlty I am doing this but...
- 3236 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Ashwin_DSA Thanks for the reply.The method you proposed sounds fine, but we are dealing with a very large volume of data around 3 schemas, ~50 tenants, and over 100 tables. Since this data is being migrated from Azure to GCP, we would prefer to ...
- 1 kudos
- 1115 Views
- 3 replies
- 0 kudos
Resolved! Accessing secrets(secret scope) in pipeline yml file
How can I access secrets in pipeline yaml or directly in python script file?
- 1115 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @prakharsachan ,In Declarative Automation Bundles YAML (formerly known as Databricks Assets Bundles) you can only define secret scopes:If you want to read secrets from secret scope you can use dbutils in python script:password = dbutils.secrets.ge...
- 0 kudos
- 674 Views
- 1 replies
- 1 kudos
Data System & Architecture - PySpark Assignment
Title: Spark Structured Streaming – Airport Counts by CountryThis notebook demonstrates how to set up a Spark Structured Streaming job in Databricks Community Edition.It reads new CSV files from a Unity Catalog volume, processes them to count airport...
- 674 Views
- 1 replies
- 1 kudos
- 1 kudos
That's cool ! why not git it ?
- 1 kudos
- 2699 Views
- 6 replies
- 2 kudos
Resolved! Get task_run_id that is nested in a job_run task
Hi, I'm wondering if there is an easier way to accomplish this.I can use Dynamic Value reference to pull the run_id of Parent 1 into Parent 2, however, what I'm looking for is for Child 1's task run_id to be referenced within Parent 2.Currently I am ...
- 2699 Views
- 6 replies
- 2 kudos
- 2 kudos
Hi @ChristianRRL you're absolutely right, and I apologize for the earlier suggestion. I've verified that task values from child jobs are not propagated back through run_job tasks. Your instinct about the REST API was correct. Here's the fix: Solutio...
- 2 kudos
- 1116 Views
- 2 replies
- 2 kudos
Resolved! Get task_run_id (or job_run_id) of a *launched* job_run task
Hi there, I'm finding this a bit trickier than originally expected and am hoping someone can help me understand if I'm missing something.I have 3 jobs:One orchestrator job (tasks are type run_job)Two "Parent" jobs (tasks are type notebook)parent1 run...
- 1116 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi, I ran into the same confusion and did some testing on this. Here's what I found: Task values don't cross the run_job boundary. So even if child1 sets a task value with dbutils.jobs.taskValues.set(), the orchestrator can't read it. But {{tasks.par...
- 2 kudos
- 1681 Views
- 4 replies
- 0 kudos
Databricks file based trigger to sharepoint
Hi,Can we create a file based trigger from sharepoint location for excel files from databricks. So my need is to copy the excel files from sharepoint to external volumes in databricks so can it be done using a trigger that whenever the file drops in ...
- 1681 Views
- 4 replies
- 0 kudos
- 0 kudos
File-based triggers in Databricks are designed to work with data that already resides in cloud storage (such as ADLS, S3, or GCS). In this case, since the source system is SharePoint, expecting a native file-based trigger from Databricks is not feasi...
- 0 kudos
- 3190 Views
- 3 replies
- 1 kudos
Resolved! Metric views joins
I am currently working on a migration project from power BI to ai bi dashboard in databricks . Now i am using the metric views to create all the measures and DAX queries which i have in my power BI report in YAML in the metric views but the main prob...
- 3190 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey @Akshatkumar69, welcome to the community. You're not alone on this one, it is common with folks coming from Power BI. The key thing to understand is that AI/BI charts do expect a single data source, but that source can be a metric view that alrea...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
