- 3098 Views
- 2 replies
- 0 kudos
Materialized Views Incremental Load
My question is: can materialized views be updated incrementally. For example:In my case, we store the data in Iron layer and it gets flattened in bronze and silver layer as separate tables. The required transformations happen from silver to gold laye...
- 3098 Views
- 2 replies
- 0 kudos
- 0 kudos
So, I am using DLT declarative framework to work with MV in the gold layer. I am sharing code for the sample. So, can someone tell me that how we can do only the incremental refresh, as this code is doing a full refresh daily. I don't to want do agai...
- 0 kudos
- 4579 Views
- 4 replies
- 1 kudos
Incremental refresh of materialized view in serverless DLT
Hello, Every time that I run a delta live table materialized view in serverless , I get a log of "COMPLETE RECOMPUTE" . How can I achieve incremental refresh in serverless in DLT pipelines?
- 4579 Views
- 4 replies
- 1 kudos
- 1 kudos
So, I am using DLT declarative framework to work with MV in the gold layer. I am sharing code for the sample. So, can someone tell me that how we can do only the incremental refresh, as this code is doing a full refresh daily. I don't to want do agai...
- 1 kudos
- 2929 Views
- 4 replies
- 6 kudos
Resolved! Databricks clusters unresponsive
Hello everyone,we are experiencing issues on one of our Databricks workspaces:Notebooks and SQL queries are executing, but results are not returned to the UI.On the screenshots you can see examples where cells in notebooks and queries in a SQL wareho...
- 2929 Views
- 4 replies
- 6 kudos
- 6 kudos
Thanks a lot for the quick replies!It seems the issue was related to a faulty iPython version (or something similar inside the workspace). The problem resolved itself later the same day in the evening, and everything is working normally again now.
- 6 kudos
- 6451 Views
- 11 replies
- 2 kudos
Resolved! Issue with Installing Remorph Reconcile Tool and Compatibility Clarification
I am currently working on a table migration project from a source Hive Metastore workspace to a target Unity Catalog workspace. After migrating the tables, I intend to write table validation scripts using the Remorph Reconcile tool. However, I am enc...
- 6451 Views
- 11 replies
- 2 kudos
- 2 kudos
I would just like to mention that it might not be due to remorph or your python version that you encounter this error. I got a similar error message when trying to apply changes to the metastore grants through terraform.It worked when I authenticate...
- 2 kudos
- 1271 Views
- 1 replies
- 2 kudos
Resolved! New Scroll bar appears in cells with more than 300 lines
Hi,In a Databricks notebook, I noticed a new behaviour which I'd like to deactivate.Essentially, when a cell reaches 300 lines or more, a new scroll bar within the cell appears. I'd rather have the cell displayed in full and keep using the page scrol...
- 1271 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @ToNiOZ45 ,Unfortunately, there's no option to disable it in workspace settings. You can create ticket in Databricks Ideas and suggest them to add that option if you find this scroll bar frustrating:Feedback | Databricks
- 2 kudos
- 2172 Views
- 1 replies
- 0 kudos
Deploy asset bundle without recreating Lakeflow SQL gateway and DLT Pipeline
Hello all,I have a pre-existing Databricks Asset Bundle that deploys a workflow.I am starting to investigate Lakeflow Connect using a SQL Server connector. In my existing Databricks Asset Bundle, I added a pipeline YAML file that defines a gateway an...
- 2172 Views
- 1 replies
- 0 kudos
- 0 kudos
@LJacobsen If i understand you correctly, you have manually created objects (in your case a DLT pipeline) that you now want to manage in your asset bundle. If that's the case "bundle deployment bind" is your friend:https://learn.microsoft.com/en-us/a...
- 0 kudos
- 2851 Views
- 2 replies
- 0 kudos
Executors Getting FORCE_KILL After Migration to GCE – Resource Scaling Not Helping
Hi everyone,We're facing a persistent issue with our production streaming pipelines where executors are being forcefully killed with the following error:Executor got terminated abnormally due to FORCE_KILL Screenshot for reference:Context:Our pipelin...
- 2851 Views
- 2 replies
- 0 kudos
- 0 kudos
@minhhung0507 wrote:We're facing a persistent issue with our production streaming pipelines where executors are being forcefully killed with the following error:Executor got terminated abnormally due to FORCE_KILLI solved the issue in our case and al...
- 0 kudos
- 650 Views
- 1 replies
- 1 kudos
connect RabbitMQ hosted on AWS EC2 instance
We are facing an issue. We created a cluster "dev_cluster_catalog_support" . When we are using compute policy "unrestricted" we are able to connect RabbitMQ hosted on AWS EC2 instance, but when we change the compute policy "Shared Compute" getting er...
- 650 Views
- 1 replies
- 1 kudos
- 1 kudos
RabbitMQ, it brings me back to my IoT days With that said, here is some helpful guidance: The issue occurs because Shared Compute cluster policies enforce stricter network access controls compared to Unrestricted policies. When using the Unrestricte...
- 1 kudos
- 1918 Views
- 3 replies
- 0 kudos
Ingest data from SQL Server
I've been working on data ingestion from SQL Server to UC using lakeflow connect. Lakeflow connect actually made the work easier when everything is right. I am trying to incorporate this with DAB and this would work fine with schema and table tags fo...
- 1918 Views
- 3 replies
- 0 kudos
- 0 kudos
I’ve been using the notebook style as there are some transformations that need to be done on destination_table names as the source_table names has special characters. My requirement is: for example, I have a schema with 5 tables, 10 individual tables...
- 0 kudos
- 846 Views
- 3 replies
- 0 kudos
Databricks Row filter causing duplicates in Delta Table while writing
Hello,I have a databricks external table to which I have applied a row filter, After applying the row filter I see that the Delta merge functionality is not working as expected even with simple records ( not nested) and I have already tried execution...
- 846 Views
- 3 replies
- 0 kudos
- 0 kudos
What is the workaround to achieve the same functionality of upsert that we get with Merge?
- 0 kudos
- 1490 Views
- 2 replies
- 3 kudos
Run failed with error message Failed to resolve references: Parameter values exceed the size limit
Hi community,My team and I are facing an issue with the Parameter Values (see the title of this discussion) being passed through each task of a job. Unfortunately, this causes our job run to fail.Do you have any suggestions on how to handle parameter...
- 1490 Views
- 2 replies
- 3 kudos
- 3 kudos
We're dealing with this issue on our project in following way:- we have defined config JSON file (could also be YAML - doesn't matter)- and now let's say that you have param that has really long value - for the sake of example let's consider paramete...
- 3 kudos
- 2104 Views
- 3 replies
- 0 kudos
Azure Databricks – Lakehouse Federation MySQL Connection Fails but Works in Notebook
I am trying to set up a Lakehouse Federation connection to an Azure MySQL database. When I connect from a Databricks notebook using Python - mysql.connector, same cluster, it works fine.But when I set up the Lakehouse Federation connection and test i...
- 2104 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @asrivas , I’ve been trying to simulate this on my side and in my case I was able to complete the connection, but I believe in your case the issue comes from the MySQL setting --require_secure_transport=ON. In the notebook it works because the dri...
- 0 kudos
- 3924 Views
- 7 replies
- 10 kudos
Resolved! How to overwritten job parameter task inside a job task
Hi community,How to overwritten the job parameter inside the job task? Because seems that the job parameter has a higher priority than a task parameter although it is overwritten
- 3924 Views
- 7 replies
- 10 kudos
- 10 kudos
Hi Pilsner,Thanks for your response, the issue is that I need to know it before. In this case, we need to set inside a notebook for example the task values. I want to be able to set it at task value. I think it is not provided from Databricks.
- 10 kudos
- 1323 Views
- 1 replies
- 0 kudos
DLT Pipeline from Streaming Table
HiI have a bronze table with Product_id, *, start_at, end_At which is a streaming and SCD Type 2 Table, which means any change in product_attributes would insert a new row with end_at as null. So when we take this table with end_at as null , the tabl...
- 1323 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @km1837 ,Instead of trying to implement a stream table on a stream table, for your use case I think using Materialized View on next child table would be the best choice.For e.g.: @dlt.table(name="workspace.silver.current_product") def sample_trips...
- 0 kudos
- 2932 Views
- 9 replies
- 3 kudos
Resolved! issue updating DLT pipeline configurations using databricks sdk
I am updating dlt pipeline configs with job id , run id and run_datetime of the job , so that i can access these values inside dlt pipeline. below is the code i am using to do that. # Databricks notebook sourceimport sysimport loggingfrom databricks....
- 2932 Views
- 9 replies
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 580 | |
| 349 | |
| 287 |
