- 19139 Views
- 27 replies
- 4 kudos
spotify API get token - raw_input was called, but this frontend does not support input requests.
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
- 19139 Views
- 27 replies
- 4 kudos
- 4 kudos
I really enjoyed reading it and found the insights both practical and engaging. It's always great to come across content that provides real value to readers. I also enjoy exploring articles on digital trends, lifestyle, and informative topics at Con...
- 4 kudos
- 25 Views
- 0 replies
- 0 kudos
from_utc_timestamp silently double-shifts time when session timezone isn't UTC — docs should call th
Summary:from_utc_timestamp(current_timestamp(), '<tz>') produces an incorrect (future-shifted) timestamp whenever the Spark session timezone is already set to something other than UTC. This is a very common pattern for teams stamping dp_load_ts/creat...
- 25 Views
- 0 replies
- 0 kudos
- 95 Views
- 1 replies
- 1 kudos
Lakebase CDF — destination Delta table not created after successful UI setup (Free Edition)
Summary:We configured Lakebase CDF to stream changes from a native Postgres table (public.query_logs) into Unity Catalog, but the destination Delta table never appears despite completing all documented prerequisites.Steps taken1. Created query_logs ...
- 95 Views
- 1 replies
- 1 kudos
- 1 kudos
Most likely this is a Free Edition storage limitation rather than a bug, and your Postgres-side setup sounds correct. Lakebase CDF writes its destination as a Unity Catalog managed Delta table, and the Lakebase CDF docs list catalogs backed by defaul...
- 1 kudos
- 13320 Views
- 7 replies
- 2 kudos
Relevance of off heap memory and usage
I was referring to the doc - https://kb.databricks.com/clusters/spark-executor-memory.In general total off heap memory is = spark.executor.memoryOverhead + spark.offHeap.size. The off-heap mode is controlled by the properties spark.memory.offHeap....
- 13320 Views
- 7 replies
- 2 kudos
- 168 Views
- 5 replies
- 0 kudos
Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data
Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data, same works fine with normal compute storage and external location, credentials, IAM policy and serverless egress rules all seems to be correct because normal comp...
- 168 Views
- 5 replies
- 0 kudos
- 0 kudos
Your read is right, and fair point on the allowlist: editing network policies is part of serverless egress control, which is an Enterprise capability, so on Premium the propagation window is the path. Your timing hypothesis matches how this works int...
- 0 kudos
- 712 Views
- 4 replies
- 3 kudos
Is it required to run Lakeflow Connect on Serverless?
As the subject states, my question is:Is it required to run the Ingestion Pipeline in Lakeflow Connect on Serverless compute? Cause I try to define my own cluster in the DAB, but it raises an error:`Error: cannot create pipeline: You cannot provide c...
- 712 Views
- 4 replies
- 3 kudos
- 3 kudos
Yes — Lakeflow Connect ingestion pipelines always run on Serverless compute. Databricks overrides your compute config and switches back to serverless,because the ingestion connector requires it.
- 3 kudos
- 99 Views
- 0 replies
- 0 kudos
Bundle Validation Error: Volume lifecycle Field Rejected Despite Being in Schema
Environment:Bundle engine: directResource type: Volume (Unity Catalog)Problem:I'm trying to use the lifecycle field with prevent_destroy: true on a Unity Catalog volume resource in my Declarative Automation Bundle (DAB). the validation fails with:Err...
- 99 Views
- 0 replies
- 0 kudos
- 228 Views
- 0 replies
- 0 kudos
How to Prepare Enterprise Data for AI Agents: A Practical Guide
Artificial intelligence has moved beyond chatbots and predictive models. Today, enterprises are adopting AI agents that can retrieve information, analyze business data, automate workflows, and support decision-making across departments.The success of...
- 228 Views
- 0 replies
- 0 kudos
- 581 Views
- 7 replies
- 1 kudos
Resolved! DBR 17.3 – sporadic import failures from bundle
Hi all,Since two days ago we've been getting sporadic Python import failures on DBR 17.3, with no change on our side. The errors are either "cannot import name ..." or "no module named ...", and they come and go: the same code fails on one run and pa...
- 581 Views
- 7 replies
- 1 kudos
- 1 kudos
Thank you for your reply, we have already filed a support ticket. We might look into packaging our code as wheel if there is no other way.
- 1 kudos
- 1758 Views
- 6 replies
- 5 kudos
Resolved! Intermittent failure with Python IMPORTS statements after upgrading to DBR18.0
We have a python module (WidgetUtil.py) that sits in the same folder as our notebook. For the past few years we have been using a simple import statement to use it. Starting with DBR18.0 the imports fails intermittently (25% of the time) when running...
- 1758 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi, are you aware if this is also affecting Databricks Free Edition environment v5 ? I think I am having the same issue.Thanks.
- 5 kudos
- 167 Views
- 1 replies
- 0 kudos
Intialization stage is taking time in lakehouse pipeline
Hello Team,My intialization stage in lakehouse pipeline is taking 5-6 mins. When i checked event log table,below are statsMessage Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'. is repeatedly coming for 4-5 mins....
- 167 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Anish_2, The message... Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'..is usually not enough on its own to identify the root cause. I would treat it as a metrics/observability signal rather than the primary ...
- 0 kudos
- 1995 Views
- 8 replies
- 0 kudos
Disable the dashboard sharing field for dashboard creators
"How can I disable the dashboard sharing field for dashboard creators who are not admins? I tried changing the creator’s permission from 'CAN_MANAGE' to 'CAN_READ', but it had no effect — the creator still retains the 'CAN_MANAGE' permission
- 1995 Views
- 8 replies
- 0 kudos
- 0 kudos
Hey, that thread's already got the core answer right: there's no setting to strip CAN_MANAGE from a dashboard creator, and you already found that out the hard way when the API downgrade to CAN_READ had no effect. That's expected, Databricks docs conf...
- 0 kudos
- 218 Views
- 1 replies
- 1 kudos
Resolved! Delta Table in DLT pipeline
Hi all, Is it possible to create a Delta table using DLT pipeline. I'm able to create a Delta table using Job and also able to create Materialized view and Streaming table using DLT pipleines. But I want to create Delta table using DLT pipelines. Is ...
- 218 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, yes, both streaming tables and materialized views in a DLT pipeline (now called Lakeflow Declarative Pipelines) are Delta tables under the hood. There isn't a separate "plain Delta table" object type inside a pipeline, streaming tables and materi...
- 1 kudos
- 5317 Views
- 12 replies
- 5 kudos
How to find the source code for the data engineering learning path?
Hi Everyone,I am taking data engineering learning path in customer-academy.databricks.com . I am not able to find any source code attached to the course. Can you please help me to find it so that I can try hands on as well ?ThanksRupesh
- 5317 Views
- 12 replies
- 5 kudos
- 5 kudos
The last module in "Get started with data engineering" learning path is wrong.Please find the screenshot attached. In the video the file used was different that the one present in resource section. Also i think in the video they were using file renam...
- 5 kudos
- 187 Views
- 2 replies
- 0 kudos
lakeflow connect postgres sql
Hi can anyone tell me how to enable tc LakeFlow Connect for Direct Cdc Managed Ingestion Pipeline. it shows that this mode is not available for this workspace for the selected connector.
- 187 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @tasa, Just responded to your other post on the same topic. It looks like your workspace is not currently enabled for Direct CDC/integrated CDC for that connector. In Databricks, this commonly surfaces as the DIRECT_CDC_NOT_ENABLED valid...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 576 | |
| 349 | |
| 287 |
