Hi everyone,I recently dealt with a frustrating scenario: a Databricks job that usually takes minutes ran for 18 hours without failing, quietly consuming compute and blocking downstream pipelines.The driver hadn't crashed, and the job hadn't failed—i...
Hi @Ashwin_DSA Thanks for the response ‌‌I was thinking if the results are precomputed using cube and persisted using mv that would be retrieve results faster than metric view. Could u please let me know if my understanding is correct
Hi everyone,I am seeing an unexpected behavior with Auto Loader when using Managed File Events on Classic Compute.The error message itself seems inconsistent with the behavior I am seeing:[FAILED_READ_FILE.DBR_FILE_NOT_EXIST] Error while reading file...
Just sharing a solution in case anyone runs into the same issue.The error was caused by the cluster configuration including spark.master: "local[*]". After removing this setting, the error stopped occurring and the Auto Loader finished correctly.This...
Today's digital learning environment depends on Learning Management Systems as crucial components. The system facilitates effective knowledge distribution while educational institutions use it to organize learning activities. LMS platforms will conti...
Has anyone else run into a situation where a breaking schema change on a SQL Server source table leaves their Lakeflow Connect pipeline in a state it can't recover from — even after destroying and recreating the pipeline?Here's what happened to us:- ...
Hi @lrm_data yes, this one catches a lot of people. A few things to check on the SQL Server side that commonly block recovery even after destroy + recreate:Stale lakeflow_* capture instance. SQL Server allows only 2 capture instances per table. If bo...
I have a Databricks workspace on AWS (serverless compute). I created a network policy with "Allow access to all destinations" enabled and attached it to my workspace. When I run a Python notebook and try to make an HTTP request or curl to any externa...
Most likely the egress policy change hasn’t actually taken effect on the serverless compute that’s running your notebook. Check these things in order: Verify the network policy itself (Account Console → Security → Networking → Context-based ingress ...
Hey All!We are trying out the Beta connector for SharePoint and found that the connector will not work at the root-level site. Is there a reason for this limitation. It is unfortunately a hard blocker for us to use the native connector. MUST_START...
Is there any documentation available around the changeset size thresholds for materialized view incremental refreshes? Are these configurable at all? Are they constant or do the thresholds change depending on the number of rows/size of the material...
Hi, On top of Pradeep's reply, which I'd recommend trying, I'd also suggest you raise a support ticket for this. They will potentially be able to tweak the settings in the backend (not guaranteed), but it may help. Thanks,Emma
Has anyone else seen full refresh snapshots trigger outside of their configured refresh window in Lakeflow Connect?Here's our situation:- We have a full refresh window configured to restrict snapshot operations to off-hours- On at least one occasion,...
@lrm_data This is very unlike case for the refresh to be triggered outside the configured window. Though I would still suggest to check the Configured Window and Auto Full Refresh policy once to be sure.If still persists, then you may raise a support...
Issue Description:I am attempting to disable public network access on the Azure Databricks managed storage account. However, I am encountering the following error:Failed to save resource settings — access is denied due to a deny assignment created by...
@MyProfile This would be helpful, check once - https://learn.microsoft.com/en-us/answers/questions/1707749/managed-storage-accounts-compliance
Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the ...
My Blog on thishttps://medium.com/@singh.sanjiv/truncate-and-load-streaming-live-table-8f840eb424d1
Hi,we're exploring replacing one of the use cases we are running in our clour provider with a Databricks pipelines. We currently have explored possibility to subscribe to an eventhub using SDP pipelines, feedding our iot data into a Delta table where...
Hi @leopold_cudzik, The pattern you are suggesting is feasible, but it’s much easier to manage if you separate history ingestion from the 7-day serving view instead of cleaning the streaming sink table in place. A common architecture on Databricks wo...
Queries that previously worked started failing in SQL Warehouse (Dashboards) without any changes on our side.The query succeeds, but fails to render results with error:"Cannot read properties of undefined (reading 'data')"This happens with:- system.b...
Same problem here. I have previously reported this issue, and it had been resolved at the time. However, the problem has now reoccurred.When ingesting large tables (over 100k rows), the system is unable to properly render the data, preventing the tab...
Hi Databricks Community,We are looking for the recommended best practice for managing environment-specific configurations (catalogs and schemas) within our SQL files when using Databricks Asset Bundles.Our Setup:We use Databricks Asset Bundles to pac...
Hi @DineshOjha The best approach is parameterized SQL with widget-based defaults in your Python wrapper, wired to DABs target variables. Why this works on both fronts: Engineers run the notebook interactively and widget defaults kick in (dev values)....
Hi,I have ran into a problem when creating materialized view.Here's my simple query I'm trying to run:%sql create or replace materialized view catalog.schema.mView_test as select * from catalog.schema.table limit 10;I'm getting following error:Encoun...
There are multiple requirements for materialized views. You can check belowYou must use a Unity Catalog enabled pro or serverless SQL warehouse.To incrementally refresh a materialized view from Delta tables, the source tables must have row tracking e...
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 555 | |
| 349 | |
| 287 |
