- 4941 Views
- 4 replies
- 1 kudos
Dangerous implicit type conversions on 17.3 LTS.
Starting with DBR 17 running Spark 4.0, spark.sql.ansi.enabled is set to true by default. With the flag enabled, strings are implicitly converted to numbers in a very dangerous manner. ConsiderSELECT 123='123';SELECT 123='123X';The first one is succe...
- 4941 Views
- 4 replies
- 1 kudos
- 1 kudos
FYI, it seems I was mistaken about the behaviour of '::' on Spark 4.0.1. It does indeed work like CAST on both DBR 17.3 and Spark 4.0.1 and raises an exception on '123X'::int. The '?::' operator seems to be a Databricks only extension at the moment (...
- 1 kudos
- 1239 Views
- 2 replies
- 3 kudos
Resolved! Using Init Scipt to execute python notebook at all-purpose cluster level
HiWe have setup.py in my databricks workspace.This script is executed in other transformation scripts using%run /Workspace/Common/setup.pywhich consume lot of time. This setup.py internally calls other utilities notebooks using %run%run /Workspace/Co...
- 1239 Views
- 2 replies
- 3 kudos
- 3 kudos
You can’t “%run a notebook” from a cluster init script—init scripts are shell-only and meant for environment setup (install libs, set env vars), not for executing notebooks or sharing Python state across sessions. +1 to what @Raman_Unifeye has told. ...
- 3 kudos
- 705 Views
- 1 replies
- 2 kudos
Notebooks suddenly fails to retrieve Databricks secrets
At around 5:30 am (UTC+11) this morning, a number of our scheduled serverless notebook jobs started failing when attempting to retrieve Databricks secrets.We are able to retrieve the secrets using the databricks CLI and the jobs are run as a user tha...
- 705 Views
- 1 replies
- 2 kudos
- 2 kudos
me tooBut it looks like there hasn't been any official reply regarding this matter yet.
- 2 kudos
- 1227 Views
- 3 replies
- 0 kudos
Connecting to an S3 compatible bucket
Hi everyone,I’m trying to connect Databricks to an S3-compatible bucket using a custom endpoint URL and access keys.I’m using an Express account with Serverless SQL Warehouses, but the only external storage options I see are AWS IAM roles or Cloudfla...
- 1227 Views
- 3 replies
- 0 kudos
- 0 kudos
Serverless compute does not support setting most Apache Spark configuration properties irrespective of Enterprise Tier as dB fully manages the underlying infrastructure.
- 0 kudos
- 1907 Views
- 3 replies
- 4 kudos
Resolved! What's the difference between dbmanagedidentity and a storage credential based on managed identity?
I’m looking for guidance on the differences between:dbmanagedidentity (the workspace-managed identity), andUnity Catalog storage credentials based on Azure Managed IdentitySpecifically, I’d like to understand:What are the key differences between thes...
- 1907 Views
- 3 replies
- 4 kudos
- 4 kudos
use dbmanageidentity for non‑storage Azure services, such as Cosmos DB, Azure SQL, Event Hub, Key vault.
- 4 kudos
- 2019 Views
- 5 replies
- 6 kudos
Self-referential foreign key constraint for streaming tables
When defining a streaming tables using DLT (declarative pipelines), we can provide a schema which lets us define primary and foreign key constraints.However, references to self, i.e. the defining table, are not currently allowed (you get a "table not...
- 2019 Views
- 5 replies
- 6 kudos
- 6 kudos
Each of these workarounds give up the optimizations that are enabled by the use of key constraints.
- 6 kudos
- 850 Views
- 1 replies
- 0 kudos
Databricks pipeline fails expectation on execute python script, throws error: Update FAILES
Hi Community,I'm new to Databricks and am trying to make and implement pipeline expectations, The pipelines work without errors and my job works. I've tried multiple ways to implement expectations, sql and python. I keep resolving the errors but end ...
- 850 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey, I think it may be the row_count condition causing the issue. The expectation runs on each row and sees if the record meets the criteria in the expectation, so you're effectively asking count * on each record, which will always evaluate to 1 and...
- 0 kudos
- 1724 Views
- 1 replies
- 1 kudos
Resolved! How to stop Databricks retaining widget selection between runs?
I have a Python notebook in Databricks. Within it I have a multiselect widget, which is defined like this:widget_values = spark.sql(f''' SELECT my_column FROM my_table GROUP BY my_column ORDER BY my_column ''') widget_values = widget_values.collect(...
- 1724 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @SRJDB , What you’re running into isn’t your Python variable misbehaving—it’s the widget hanging onto its own internal state. A Databricks widget will happily keep whatever value you gave it, per user and per notebook, until you explicitly clea...
- 1 kudos
- 1457 Views
- 1 replies
- 1 kudos
Resolved! DLT quota Issue
Hi Community,I am facing a weird problem within my Azure Databricks workspace, I am trying to create and run SDP, but somehow when I try to run more than 1 pipeline in parallel, it gives me an error (pasting the error message below). I currently only...
- 1457 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @AyushPaldecha09! This error usually appears due to concurrency limits. If you're already on a Premium tier, you typically shouldn’t be hitting this cap, so the best next step is to open a Databricks Support ticket and request an increase to yo...
- 1 kudos
- 2575 Views
- 4 replies
- 2 kudos
Time Travel Error when selecting from materialized view (Azure Databricks)
Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...
- 2575 Views
- 4 replies
- 2 kudos
- 2 kudos
OUR FIXEventually we learned via DESCRIBE EXTENDED, that the table had lost its Location path in production environment. Querying via the development environment to the production catalog the location path was still visible.Solved the issue (with t...
- 2 kudos
- 2885 Views
- 3 replies
- 4 kudos
Resolved! Workflow scheduling on particular working day of the month in ADB
Hi,I am looking to schedule a workflow to execute on 3rd working day. Working day here would be Mon-Fri of each month. I could not find any direct crontab solution but have created watcher file solution for it. Below code will create a watcher file a...
- 2885 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @bianca_unifeye , @Poorva21 ,Thank you for your valuable responses.I have changed the main workflow trigger as file arrival in scheduling. I have scheduled the watcher file code to run daily in weekdays and as soon as the 3rd working day encounter...
- 4 kudos
- 3112 Views
- 1 replies
- 0 kudos
Resolved! Issue Genie API - different responses in UI and via API
Hi community,I created an agent with a genie tool, a wrapper around a GenieAgent connected to my Genie space (GENIE_SPACE_ID) that sends user questions and returns Genie’s textual response. I noticed I get 02 different responses when I post a questio...
- 3112 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @D4F , What you’re seeing is normal behavior—and the good news is there are very real, very practical ways to make your Genie-based agent more consistent without resorting to a giant, brittle prompt. Let’s dig in. First, why the UI and API can r...
- 0 kudos
- 5334 Views
- 4 replies
- 3 kudos
Databricks asset bundle occasionally duplicating jobs
Since last year, we have adopted Databricks Asset Bundles for deploying our workflows to the production and staging environments. The tool has proven to be quite effective, and we currently use Azure DevOps Pipelines to automate bundle deployment, tr...
- 5334 Views
- 4 replies
- 3 kudos
- 6910 Views
- 4 replies
- 3 kudos
Delta Tables: Time-To-Live
I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. Do...
- 6910 Views
- 4 replies
- 3 kudos
- 5590 Views
- 6 replies
- 1 kudos
Resolved! AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE when starting SQL Server Ingestion pipeline
Dear Community,I’m seeing a compute error when running a Databricks ingestion pipeline (Lakeflow managed ingestion) on AWS.Cloud : AWSRegion: ap−northeast−2Source: SQL Server ingestion pipelineWhen I start the ingestion pipeline, it fails with the f...
- 5590 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi, I'm afraid you cannot edit compute instance type settings for SQL Server ingestion pipelines via the Databricks UI. Such changes can only be made via API.
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
