AI only matters when it improves workflows, speeds up decisions, and creates measurable business value.So, what would real AI impact look like inside your company?
Summary:from_utc_timestamp(current_timestamp(), '<tz>') produces an incorrect (future-shifted) timestamp whenever the Spark session timezone is already set to something other than UTC. This is a very common pattern for teams stamping dp_load_ts/creat...
Summary:We configured Lakebase CDF to stream changes from a native Postgres table (public.query_logs) into Unity Catalog, but the destination Delta table never appears despite completing all documented prerequisites.Steps taken1. Created query_logs ...
Most likely this is a Free Edition storage limitation rather than a bug, and your Postgres-side setup sounds correct. Lakebase CDF writes its destination as a Unity Catalog managed Delta table, and the Lakebase CDF docs list catalogs backed by defaul...
I was referring to the doc - https://kb.databricks.com/clusters/spark-executor-memory.In general total off heap memory is = spark.executor.memoryOverhead + spark.offHeap.size. The off-heap mode is controlled by the properties spark.memory.offHeap....
As the subject states, my question is:Is it required to run the Ingestion Pipeline in Lakeflow Connect on Serverless compute? Cause I try to define my own cluster in the DAB, but it raises an error:`Error: cannot create pipeline: You cannot provide c...
Yes — Lakeflow Connect ingestion pipelines always run on Serverless compute. Databricks overrides your compute config and switches back to serverless,because the ingestion connector requires it.
Hi all,Since two days ago we've been getting sporadic Python import failures on DBR 17.3, with no change on our side. The errors are either "cannot import name ..." or "no module named ...", and they come and go: the same code fails on one run and pa...
Thank you for your reply, we have already filed a support ticket. We might look into packaging our code as wheel if there is no other way.
We have a python module (WidgetUtil.py) that sits in the same folder as our notebook. For the past few years we have been using a simple import statement to use it. Starting with DBR18.0 the imports fails intermittently (25% of the time) when running...
Hi, are you aware if this is also affecting Databricks Free Edition environment v5 ? I think I am having the same issue.Thanks.
Hello Team,My intialization stage in lakehouse pipeline is taking 5-6 mins. When i checked event log table,below are statsMessage Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'. is repeatedly coming for 4-5 mins....
Hi @Anish_2, The message... Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'..is usually not enough on its own to identify the root cause. I would treat it as a metrics/observability signal rather than the primary ...
"How can I disable the dashboard sharing field for dashboard creators who are not admins? I tried changing the creator’s permission from 'CAN_MANAGE' to 'CAN_READ', but it had no effect — the creator still retains the 'CAN_MANAGE' permission
Hey, that thread's already got the core answer right: there's no setting to strip CAN_MANAGE from a dashboard creator, and you already found that out the hard way when the API downgrade to CAN_READ had no effect. That's expected, Databricks docs conf...
Hi all, Is it possible to create a Delta table using DLT pipeline. I'm able to create a Delta table using Job and also able to create Materialized view and Streaming table using DLT pipleines. But I want to create Delta table using DLT pipelines. Is ...
Hi, yes, both streaming tables and materialized views in a DLT pipeline (now called Lakeflow Declarative Pipelines) are Delta tables under the hood. There isn't a separate "plain Delta table" object type inside a pipeline, streaming tables and materi...
Hi Everyone,I am taking data engineering learning path in customer-academy.databricks.com . I am not able to find any source code attached to the course. Can you please help me to find it so that I can try hands on as well ?ThanksRupesh
The last module in "Get started with data engineering" learning path is wrong.Please find the screenshot attached. In the video the file used was different that the one present in resource section. Also i think in the video they were using file renam...
Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.
Hi @tasa, Just responded to your other post on the same topic. It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED valid...
Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.
Hi @tasa, It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED validation error, which means the feature is not enabled for the org...
Hi Expert,when global filter page is added then visual header alignement gets missedThoughts: The user is reporting a UI/layout issue with the Lakeview dashboard — when a global filter page is added, the visual header alignment gets misaligned. This ...
Hi @Rameshw, As you have pointed out, what you’re seeing is expected behaviour when a filter is added above an existing grid layout. Widgets on the canvas retain their existing positions, so adding a new row at the top does not automatically reflow t...
Hi,I want to install Databricks Lakebridge on my Databricks environment and use the analyze and transpile commands through a python script.The usecase is that we need to create an automated pipeline that will migrate the existing scripts from snowfla...
Hi, the error you're seeing is consistent with the execution environment rather than with Lakebridge itself. Lakebridge is a Databricks CLI Labs extension (databricks labs install lakebridge) designed to run wherever the Databricks CLI is installed—y...
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
