Hi team,Im using Databricks SDK for python to run SQL queries. I created a variable as below:param = [{'name' : 'a', 'value' :x'}, {'name' : 'b', 'value' : 'y'}]and passed it the statement as below_ = w.statement_execution.execute_statement( warehous...
@Kaniz This does not help resolve the issue. I am experiencing the same issue when following the above pointers. Here is the statement:response = w.statement_execution.execute_statement(
statement='ALTER TABLE users ALTER COLUMN :col_name SET NOT...
Hi,I'm running shallow clone for external delta tables. The shallow clone is failing for source tables where I don't have MODIFY permission. I'm getting below exception. I don't understand why MODIFY permission to source table is required. Is there a...
Also check this documentation on access mode :Shallow clone for Unity Catalog tables | Databricks on AWS Working with Unity Catalog shallow clones in Single User access mode, you must have permissions on the resources for the cloned table source as w...
Hi,I want to run a md5 checksum on the uploaded file to databricks. I can generate md5 on the local file but how do I generate one on uploaded file on databricks using CLI (Command line interface). Any help would be appreciated.I tried running databr...
Library installation failed for library due to user error for jar: \"dbfs:////<<PATH>>/jackson-annotations-2.16.1.jar\"\n Error messages:\nLibrary installation attempted on the driver node of cluster <<clusterId>> and failed. Please refer to the foll...
@pavansharma36 Thanks for the details.I had a look on runtime and platform release notes and I can't find nothing that could explain a change of behavior. I can only suppose that background changes happened but guessing is not fact.It's only an opini...
What is the fundamental difference between Fixed interval micro-batches and AvailableNow Trigger, given that both can consume in micro-batch based on the desire size of the micro batch ? Is the fundamental difference the fact that AvailableNow shut d...
Hi @Maatari,
Fixed Interval Micro-batches are like clockwork, processing data at regular intervals without stopping.AvailableNow Trigger is more adaptive, consuming data as it becomes available and then gracefully shutting down.
Regarding your confu...
We are doing a first time implementation of data streaming reading from a partitioned pulsar topics to a delta table managed by UC. We are unable to scale the job beyond about ~ 40k msgs/sec. Beyond 40k msgs/sec , the job fails. I'd imagine Databric...
Posting this here too in case anyone else has run into this issue... Trying to set up Autoloader File Notifications but keep getting an "Internal Server Error" message.Failure on Write EventSubscription - Internal error - Microsoft Q&A
I have a pyspark.pandas.frame.DataFrame object (that I called from `pandas_api` on a pyspark.sql.dataframe.DataFrame object). I have a complicated transformation that I would like to apply to this data, and in particular I would like to apply it in ...
Hi,The error indicates that the Unity Catalog does not support Spark higher-order functions, such as those used in pandas_udf. This limitation likely comes from architectural or compatibility constraints. To resolve the issue, consider alternative ap...
In the example in https://docs.databricks.com/en/delta-live-tables/expectations.html#fail-on-invalid-records, it wrote that one is able to query the DLT event log for such expectations violation. In Databricks, I can use expectation to fail or drop r...
Hi All, On Unity Catalog - what is the best way to adding members to groups using API or CLI? API should be the best option, but thought to check with you all.
I have a zipped file. It has 3 csv files. It is password protected. When I tried extracting it manually it will extract only with 7zip. I moved my zipped file to ADLS automatically and want to extract it with the password. How to unzip the file and ...
Getting below error while accessing Azure sql using jdbc from Azure databricks notebook,com.microsoft.sqlserver.jdbc.SQLServerException: Failed to authenticate the user p***** in Active Directory (Authentication=ActiveDirectoryInteractive). Unable to...
This change won't be registered as metadata thus the table will still appear on unity catalog, but you'll get an error when trying to access it, as the table metadata will point to deleted files.
Above is the screenshot of error, is there any other way of processing dbc files? Do no have access/backup to previous workspace where this code is imported from
