- 50 Views
- 2 replies
- 0 kudos
Dangerous implicit type conversions on 17.3 LTS.
Starting with DBR 17 running Spark 4.0, spark.sql.ansi.enabled is set to true by default. With the flag enabled, strings are implicitly converted to numbers in a very dangerous manner. ConsiderSELECT 123='123';SELECT 123='123X';The first one is succe...
- 50 Views
- 2 replies
- 0 kudos
- 0 kudos
The flag is enabled in Spark 4.0.1. Spark documentation also states that it's enabled by default starting with Spark 4, so presumably it's going to be enabled in future releases too. Databricks documentation says the same for DBRs starting from 17, h...
- 0 kudos
- 137 Views
- 3 replies
- 3 kudos
Workflow scheduling on particular working day of the month in ADB
Hi,I am looking to schedule a workflow to execute on 3rd working day. Working day here would be Mon-Fri of each month. I could not find any direct crontab solution but have created watcher file solution for it. Below code will create a watcher file a...
- 137 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @bianca_unifeye , @Poorva21 ,Thank you for your valuable responses.I have changed the main workflow trigger as file arrival in scheduling. I have scheduled the watcher file code to run daily in weekdays and as soon as the 3rd working day encounter...
- 3 kudos
- 12 Views
- 0 replies
- 0 kudos
DLT quota Issue
Hi Community,I am facing a weird problem within my Azure Databricks workspace, I am trying to create and run SDP, but somehow when I try to run more than 1 pipeline in parallel, it gives me an error (pasting the error message below). I currently only...
- 12 Views
- 0 replies
- 0 kudos
- 37 Views
- 0 replies
- 0 kudos
Databricks pipeline fails expectation on execute python script, throws error: Update FAILES
Hi Community,I'm new to Databricks and am trying to make and implement pipeline expectations, The pipelines work without errors and my job works. I've tried multiple ways to implement expectations, sql and python. I keep resolving the errors but end ...
- 37 Views
- 0 replies
- 0 kudos
- 34 Views
- 1 replies
- 0 kudos
Issue Genie API - different responses in UI and via API
Hi community,I created an agent with a genie tool, a wrapper around a GenieAgent connected to my Genie space (GENIE_SPACE_ID) that sends user questions and returns Genie’s textual response. I noticed I get 02 different responses when I post a questio...
- 34 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @D4F , What you’re seeing is normal behavior—and the good news is there are very real, very practical ways to make your Genie-based agent more consistent without resorting to a giant, brittle prompt. Let’s dig in. First, why the UI and API can r...
- 0 kudos
- 3092 Views
- 4 replies
- 3 kudos
Databricks asset bundle occasionally duplicating jobs
Since last year, we have adopted Databricks Asset Bundles for deploying our workflows to the production and staging environments. The tool has proven to be quite effective, and we currently use Azure DevOps Pipelines to automate bundle deployment, tr...
- 3092 Views
- 4 replies
- 3 kudos
- 3300 Views
- 4 replies
- 0 kudos
Delta Tables: Time-To-Live
I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. Do...
- 3300 Views
- 4 replies
- 0 kudos
- 130 Views
- 6 replies
- 1 kudos
AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE when starting SQL Server Ingestion pipeline
Dear Community,I’m seeing a compute error when running a Databricks ingestion pipeline (Lakeflow managed ingestion) on AWS.Cloud : AWSRegion: ap−northeast−2Source: SQL Server ingestion pipelineWhen I start the ingestion pipeline, it fails with the f...
- 130 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi, I'm afraid you cannot edit compute instance type settings for SQL Server ingestion pipelines via the Databricks UI. Such changes can only be made via API.
- 1 kudos
- 24 Views
- 0 replies
- 0 kudos
Writing to Foreign catalog
I have a running notebook job where I am doing some processing and writing the tables in a foreign catalog. It has been running successfully for about an year. The job is scheduled and runs on job cluster with DBR 16.2Recently, I had to add new noteb...
- 24 Views
- 0 replies
- 0 kudos
- 288 Views
- 3 replies
- 0 kudos
Time Travel Error when selecting from materialized view (Azure Databricks)
Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...
- 288 Views
- 3 replies
- 0 kudos
- 0 kudos
Since last monday december 8th we're experiencing this same issue: Cannot time travel Delta table to version 158. Available versions: [70, 4]. However, this is just when doing a simple SELECT * statement from a gold view (without any version specifie...
- 0 kudos
- 340 Views
- 6 replies
- 4 kudos
Resolved! Accessing Databricks data in Salesforce via zero copy
I have uploaded clickstream data as shown belowDo i have to mandatorily share via Delta sharing for values to be exposed in Salesforce ?At the Salesforce end i have confirmed that i have a working connector where i am able to see samples data , but u...
- 340 Views
- 6 replies
- 4 kudos
- 4 kudos
HI Team ,Please help me my task is to connect Databrick and salesforce data cloud with zero copy . where we need databricks data in Salesforce data cloud , also just to mention my databricks workspace + ADLS stoarge is on private end point. any hel...
- 4 kudos
- 38 Views
- 0 replies
- 0 kudos
Mongo Db connector - Connection timeout when trying to connect to AWS Document DB
I am on Databricks Run Time LTE 14.3 Spark 3.5.0 Scala 2.12 and mongodb-spark-connector_2.12:10.2.0. Trying to connect to Document DB using the connector and all I get is a connection timeout. I tried using PyMongo, which works as expected and I can ...
- 38 Views
- 0 replies
- 0 kudos
- 97 Views
- 3 replies
- 1 kudos
Resolved! Issues Creating Genie Space via API Join Specs Are Not Persisted
Hi,I’m experimenting with the new API to create a Genie Space.I’m able to successfully create the space, but the join definitions are not created, even though I’m passing a join_specs object in the same format returned by GET /spaces/{id} for an exis...
- 97 Views
- 3 replies
- 1 kudos
- 1 kudos
@Raman_Unifeye@Raman_Unifeye Thanks for the detailed explanation — that really helps clarify why my join specs weren’t being persisted.Do you know if support for persisting join_specs, sql_snippets, and measures via the API is planned for an upcoming...
- 1 kudos
- 138 Views
- 6 replies
- 3 kudos
Questions About Workers and Executors Configuration in Databricks
Hi everyone, sorry, I’m new here. I’m considering migrating to Databricks, but I need to clarify a few things first.When I define and launch an application, I see that I can specify the number of workers, and then later configure the number of execut...
- 138 Views
- 6 replies
- 3 kudos
- 3 kudos
I mean: while we’re at it @szymon_dybczak or @Raman_Unifeye , is there a place where all available Databricks configuration parameters are documented? I have some pipelines that rely on special settings, such as changing the serializer, enabling Apac...
- 3 kudos
- 169 Views
- 4 replies
- 3 kudos
IDENTIFIER in SQL Views not supported?
Dear community,We are phasing out the dollar param `${catalog_name}` because it has been deprecated since runtime 15.2.We use this parameter in many queries and should now be replaced by the IDENTIFIER clause.In the query below where we retrieve data...
- 169 Views
- 4 replies
- 3 kudos
- 3 kudos
There are two options you may want to consider:Switch to using SQL UDTFs from views in certain casesFor each session, dynamically recreate the view using CREATE VIEW via EXECUTE IMMEDIATE or via Python string templating:
- 3 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
