- 986 Views
- 1 replies
- 0 kudos
Resolved! Network Configuration
I have a Databricks workspace on AWS (serverless compute). I created a network policy with "Allow access to all destinations" enabled and attached it to my workspace. When I run a Python notebook and try to make an HTTP request or curl to any externa...
- 986 Views
- 1 replies
- 0 kudos
- 0 kudos
Most likely the egress policy change hasn’t actually taken effect on the serverless compute that’s running your notebook. Check these things in order: Verify the network policy itself (Account Console → Security → Networking → Context-based ingress ...
- 0 kudos
- 1108 Views
- 2 replies
- 1 kudos
Resolved! LDP Materialized View Incremental Refreshes - Changeset Size Thresholds
Is there any documentation available around the changeset size thresholds for materialized view incremental refreshes? Are these configurable at all? Are they constant or do the thresholds change depending on the number of rows/size of the material...
- 1108 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, On top of Pradeep's reply, which I'd recommend trying, I'd also suggest you raise a support ticket for this. They will potentially be able to tweak the settings in the backend (not guaranteed), but it may help. Thanks,Emma
- 1 kudos
- 1403 Views
- 1 replies
- 0 kudos
Disable Public Network Access on Databricks Managed Storage Account - Deny Assignment
Issue Description:I am attempting to disable public network access on the Azure Databricks managed storage account. However, I am encountering the following error:Failed to save resource settings — access is denied due to a deny assignment created by...
- 1403 Views
- 1 replies
- 0 kudos
- 0 kudos
@MyProfile This would be helpful, check once - https://learn.microsoft.com/en-us/answers/questions/1707749/managed-storage-accounts-compliance
- 0 kudos
- 6761 Views
- 5 replies
- 2 kudos
Truncate delta live table and try to repopulate it in the pipeline
Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the ...
- 6761 Views
- 5 replies
- 2 kudos
- 2 kudos
My Blog on thishttps://medium.com/@singh.sanjiv/truncate-and-load-streaming-live-table-8f840eb424d1
- 2 kudos
- 858 Views
- 1 replies
- 0 kudos
Resolved! Lakehouse sync tables over rolling history
Hi,we're exploring replacing one of the use cases we are running in our clour provider with a Databricks pipelines. We currently have explored possibility to subscribe to an eventhub using SDP pipelines, feedding our iot data into a Delta table where...
- 858 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @leopold_cudzik, The pattern you are suggesting is feasible, but it’s much easier to manage if you separate history ingestion from the 7-day serving view instead of cleaning the streaming sink table in place. A common architecture on Databricks wo...
- 0 kudos
- 1924 Views
- 6 replies
- 1 kudos
SQL Warehouse error: "Cannot read properties of undefined (reading 'data')" when querying system tab
Queries that previously worked started failing in SQL Warehouse (Dashboards) without any changes on our side.The query succeeds, but fails to render results with error:"Cannot read properties of undefined (reading 'data')"This happens with:- system.b...
- 1924 Views
- 6 replies
- 1 kudos
- 1 kudos
Same problem here. I have previously reported this issue, and it had been resolved at the time. However, the problem has now reoccurred.When ingesting large tables (over 100k rows), the system is unable to properly render the data, preventing the tab...
- 1 kudos
- 2337 Views
- 4 replies
- 0 kudos
Resolved! Materialized view creation fails
Hi,I have ran into a problem when creating materialized view.Here's my simple query I'm trying to run:%sql create or replace materialized view catalog.schema.mView_test as select * from catalog.schema.table limit 10;I'm getting following error:Encoun...
- 2337 Views
- 4 replies
- 0 kudos
- 0 kudos
There are multiple requirements for materialized views. You can check belowYou must use a Unity Catalog enabled pro or serverless SQL warehouse.To incrementally refresh a materialized view from Delta tables, the source tables must have row tracking e...
- 0 kudos
- 2517 Views
- 3 replies
- 1 kudos
How to implement MERGE operations in Lakeflow Declarative Pipelines
Hey everyone,We’ve been using Autoloader extensively for a while, and now we’re looking to transition to full Lakeflow Declarative Pipelines. From what I’ve researched, the reader part seems straightforward and clear.For the writer, I understand that...
- 2517 Views
- 3 replies
- 1 kudos
- 1 kudos
Use APPLY CHANGES INTO (SQL) or dlt.apply_changes() (Python). This is the declarative replacement for foreachBatch MERGE logic in pipelines import dlt from pyspark.sql.functions import col @dlt.table(name="bronze_events") def bronze_events(): re...
- 1 kudos
- 993 Views
- 2 replies
- 1 kudos
Resolved! pipeline config DAB
I am deploying DLT pipeline in dev environment using DABs. source code is in a python script file. In the pipeline's yml file the configuration key is set to true(with all correct indentations), yet the pipeline isnt deploying in the continuous mode....
- 993 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @prakharsachan ,Continuous must be set inside the pipeline resource definition, not under configuration.The configuration block in a SDP (former DLT) pipeline definition is for Spark/pipeline settings (key-value string pairs passed to the runtime)...
- 1 kudos
- 1375 Views
- 2 replies
- 0 kudos
Cannot create streaming table
Hi,while trying to create a streaming table in sql notebook, i am getting below error. please can you assist to fix it.The operation CREATE is not allowed: Cannot CREATE the Streaming Table `my_catalog`.`test_schema`.`emp` in Serverless Generic Compu...
- 1375 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @databrick_enthu , The error is unfortunately misleading. The Materialized View / Streaming Table on Serverless Generic Compute is not yet available for a Preview. For now, you have two possibilities: Attach the notebook to a Serverless SQL Wareh...
- 0 kudos
- 1409 Views
- 2 replies
- 0 kudos
Resolved! Bug Report: Incorrect “Next Run Time” Calculation for Interval Periodic Schedules
SummaryWhen configuring a Jobs schedule → Interval periodic trigger in Databricks, the “Next run” timestamp changes inconsistently when the interval value is modified, even though the base schedule (start time) remains the same. The next run appears ...
- 1409 Views
- 2 replies
- 0 kudos
- 0 kudos
Thanks @Ashwin_DSA for that detailed answer, that surely solves my query but I see this as a classic UX bug disguised as feature
- 0 kudos
- 815 Views
- 1 replies
- 1 kudos
Resolved! Missing workspaces in workspaces_latest but present in audit
While validating workspace coverage, we observed that some workspace_id exist in system.access.audit but return NULL in system.access.workspaces_latest. select distinct a.workspace_id, w.workspace_name from system.access.audit a left join system.acc...
- 815 Views
- 1 replies
- 1 kudos
- 1 kudos
Hey @Danish11052000, Yes... This is expected and documented behaviour, not a bug. system.access.workspaces_latest contains only active workspaces in the account. When a workspace is cancelled/removed from the account, its row is removed from this tab...
- 1 kudos
- 3102 Views
- 11 replies
- 2 kudos
Resolved! What has happened to syntax coloring for SQL queries???
What has happened to syntax coloring for SQL queries??? It seems that everything is in color blue which is confusing and hard to read the code...
- 3102 Views
- 11 replies
- 2 kudos
- 1891 Views
- 2 replies
- 2 kudos
Resolved! Redshift to Databricks Migration with Lakebridge
We are currently performing an assessment for a client’s Redshift to Databricks migration, and we would like to better understand the enhanced capabilities of Lakebridge for this use case.We would appreciate clarification on the following points:Scop...
- 1891 Views
- 2 replies
- 2 kudos
- 2 kudos
There is a nice course on Partner Academy as well . It uses SQL Server as a target system for migration but you can follow the same steps for Redshift as well . https://partner-academy.databricks.com/learn/courses/4326/lakebridge-for-sql-source-syste...
- 2 kudos
- 3520 Views
- 5 replies
- 1 kudos
Resolved! Registering Delta tables from external storage GCS , S3 , Azure Blob in Databricks Unity Catalog
Hi everyone,I am currently working on a migration project from Azure Databricks to GCP Databricks, and I need some guidance from the community on best practices around registering external Delta tables into Unity Catalog.Currenlty I am doing this but...
- 3520 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi @Ashwin_DSA Thanks for the reply.The method you proposed sounds fine, but we are dealing with a very large volume of data around 3 schemas, ~50 tenants, and over 100 tables. Since this data is being migrated from Azure to GCP, we would prefer to ...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
