I using Notebooks to do some transformations I install a new whl: %pip install --force-reinstall /Workspace/<my_lib>.whl %restart_python Then I successfully import the installed lib from my_lib.core import test However when I run my code with fo...
Hi all,we are currently running into the following issuewe are using serverless SQL warehousein a JAVA application we are using the latest Databricks JDBC driver (v2.6.36)we are querying the warehouse with a collect_list function, which should return...
Hey Wilco, The answer is no, ODBC/JDBC don't support complex types so these need to be compressed into strings over the wire (usually in JSON representation) and rehydrated on the client side into a complex object.
ERROR RetryingHMSHandler: NoSuchObjectException(message:There is no database named global_temp)should one create it in the work space manually via UI? and how?would it get overwritten if work space is created via terraform?I use 10.4 LTS runtime.
I am experiencing significant delay on my streaming. I am using changefeed connector. Its processing streaming batch very frequently but experiencing sudden halt and shows no active stage for longer time. I observed below exception continuously promp...
I'm a little confused about how streaming works with DLT. My first questions is what is the difference in behavior if you set the pipeline mode to "Continuous" but in your notebook you don't use the "streaming" prefix on table statements, and simila...
Is it possible to have custom upserts in streaming tables in a delta live tables pipeline?Use case: I am trying to maintain a valid session based on timestamp column and want to upsert to the target table.Tried going through the documentations but dl...
I am unable to execute update statements through Databricks Notebook, getting this error message "com.databricks.sql.transaction.tahoe.actions.InvalidProtocolVersionException: Delta protocol version is too new for this version of the Databricks Runti...
This is resolved, this happens when a Column in the table has a GENERATED BY DEFAULT AS IDENTITY defined. When you remove this column, it works fine
i was trying to upload data into a table in hive_metastore using SSIS using SIMBA ODBC driver. The data set is huge (1.2 million records and 20 columns) , it is taking more than 40 mins to complete. is there an config change to improve the load time.
Looks like a slow data upload into a table in hive_metastore using SSIS and the SIMBA ODBC driver. This could be due to a variety of factors, including the size of your dataset and the configuration of your system. One potential solution could be to ...
In a Databricks environment, I have cloned a repository that I have in Azure DevOps Repos, the repository is inside the path:Workspace/Repos/<user_mail>/my_repo.Then when I create a Python script that I want to call in a notebook using an import: imp...
Hi @Ramseths , If your notebook and script are in the same path, it would have picked the same relative path. Is your notebook located in /databricks/driver? Thanks!
Hi there,I want to add custom JARs to an SQL warehouse (Pro if that matters) like I can in an interactive cluster, yet I don't see a way.Is that a degraded functionality when transitioning to a SQL warehouse, or have I missed something? Thank you.
ADD JAR is a SQL syntax for Databricks runtime, it does not work for DBSQL/warehouse. DBSQL would throw this error: [NOT_SUPPORTED_WITH_DB_SQL] LIST JAR(S) is not supported on a SQL warehouse. SQLSTATE: 0A000. This feature is not supported as of now....
The following doc suggests the ability to add column comments during MV creation via the `column list` parameter.Thus, the SQL code below is expected to generate a table where the columns `col_1` and `col_2` are commented; however, this is not the ca...
@leungi you've shared the python language reference. This is the SQL Reference from where I've based my example.
Hello,I made some transform on pyspark.sql.Column object: file_path_splitted=f.split(df[filepath_col_name],'/') # return Column object file_name = file_path_splitted[f.size(file_path_splitted) - 1] # return Column object Next I used variable "file_na...
Hello @Marcin_U , Thank you for reaching out. The transformation you apply within or outside the `withColumn` method will ultimately result in the same Spark plan. The answer is no, it's not possible to have rows mismatch if you're referring to the s...
Hello, I need to migrate from Databricks Azure to AWS, using tool-databricks-migration generates many errors, if I do it manually using databeicks-cli, what would be the best practice?Any tips, for example:-first migrate notebooks-second jobs-third u...
Hi all,My terraform script fails on a databricks_grants with the error: "Error: cannot update grants: Could not find principal with name DataUsers". The principal DataUsers does not exist anymore because it has previously been deleted by terraform.Bo...
Terraform databricks provider= 1.45.0
Hi Team,My team has designed Azure Databricks solution and we are looking for solution to speed-up process.Below are details of project:1- Data is copied from SAP to ADLS-Gen-2 based External location.2- Project follows medallion architecture i.e. we...
Hi @Retired_mod , @raphaelblg , would you like to throw some light on this issue.
Library installation failed for library due to user error for jar: \"dbfs:////<<PATH>>/jackson-annotations-2.16.1.jar\"\n Error messages:\nLibrary installation attempted on the driver node of cluster <<clusterId>> and failed. Please refer to the foll...
Hi @Edouard_JH Adding more details on this issue.We faced this issue with several other jars in databricks 14.3, adding the error stacktrace for the same, seems like the error comes from changes made under https://issues.apache.org/jira/browse/SPARK-...
I want to add a column to an existing delta table with a timestamp for when the data was inserted. I know I can do this by including current_timestamp with my SQL statement that inserts into the table. Is it possible to add a column to an existing de...
Can you please provide information on the additional expenses related to using this feature compared to not utilizing it at all?
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
