- 3278 Views
- 4 replies
- 0 kudos
My code:- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source tableM...
- 3278 Views
- 4 replies
- 0 kudos
Latest Reply
Hey as previously stated you could drop the duplicates of the columns that contain the said duplicates(code you can find online pretty easily), I have had this problem myself and it came when creating a temporary view from a dataframe, the dataframe ...
3 More Replies
by
deng77
• New Contributor III
- 18734 Views
- 10 replies
- 2 kudos
I want to add a column to an existing delta table with a timestamp for when the data was inserted. I know I can do this by including current_timestamp with my SQL statement that inserts into the table. Is it possible to add a column to an existing de...
- 18734 Views
- 10 replies
- 2 kudos
Latest Reply
-- Alter the table to use the GENERATED ALWAYS functionality for the created_at column
ALTER TABLE example_table
ADD COLUMN created_at TIMESTAMP GENERATED ALWAYS AS CURRENT_TIMESTAMP();@Michael Burch​ Hi , Did you try using GENERATED ALWAYS feature. ...
9 More Replies
- 819 Views
- 2 replies
- 2 kudos
I am trying to implement a UDF for a DLT pipeline. I have seen the documentation stating that it is possible but I am getting an error after adding an SQL UDF to a cell in the notebook attached to the pipeline. The aim is to have the UDF in a separat...
- 819 Views
- 2 replies
- 2 kudos
Latest Reply
You can't. The SQL support on DLT pipeline cluster is limited compared to a normal notebook. You can still define a UDF in Python using, of course, a Python notebook. In this case, you can use the spark.sql() function to execute your original SQL cod...
1 More Replies
by
g96g
• New Contributor III
- 5814 Views
- 4 replies
- 2 kudos
what is the equivalent of "IF EXISTS" in databricks? I would like to first check something later after that use the insert into statement.
- 5814 Views
- 4 replies
- 2 kudos
Latest Reply
Is there a way to check if a table exists, without trying to drop it? something like :select table_name from system_catalogs where database_name = 'mydb' and schema_name = 'myschema' and object_name = 'mytab';
3 More Replies
by
MikeK_
• New Contributor II
- 22766 Views
- 6 replies
- 0 kudos
Hi,
I'm importing some data and stored procedures from SQL Server into databricks, I noticed that updates with joins are not supported in Spark SQL, what's the alternative I can use? Here's what I'm trying to do:
update t1
set t1.colB=CASE WHEN t2.c...
- 22766 Views
- 6 replies
- 0 kudos
Latest Reply
Hi!
This is way late, but did you ever find a solution to the CROSS APPLY-part of your question? Is it possible to do CROSS APPLY in Spark SQL, or is there something you can use instead?
5 More Replies
- 5685 Views
- 5 replies
- 1 kudos
Task: Setup connection to Azure SQL Server.A couple things have changed...*We've started using Unity Catalog, so need Unity Catalog -enabled clusters*Legacy init scripts have been deprecated, and this is how we had our pyodbc setup, etc. defined.Code...
- 5685 Views
- 5 replies
- 1 kudos
Latest Reply
Hi @js54123875 did you manage to find a solution for this? I'm facing a similar problem.ThanksDiego
4 More Replies
by
Hardy
• New Contributor III
- 2941 Views
- 4 replies
- 3 kudos
I am trying to connect to SQL through JDBC from databricks notebook. (Below is my notebook command)val df = spark.read.jdbc(jdbcUrl, "[MyTableName]", connectionProperties)
println(df.schema)When I execute this command, with DBR 10.4 LTS it works fin...
- 2941 Views
- 4 replies
- 3 kudos
Latest Reply
Try to add the following parameters to your SQL connection string. It fixed my problem for 13.X and 12.X;trustServerCertificate=true;hostNameInCertificate=*.database.windows.net;
3 More Replies
by
Mado
• Valued Contributor II
- 14531 Views
- 4 replies
- 3 kudos
I want to define a variable and use it in a query, like below: %sql
SET database_name = "marketing";
SHOW TABLES in '${database_name}';However, I get the following error:ParseException:
[PARSE_SYNTAX_ERROR] Syntax error at or near ''''(line 1, pos...
- 14531 Views
- 4 replies
- 3 kudos
Latest Reply
Another option is demonstrated by this example:%sql
SET database_name.var = marketing;
SHOW TABLES in ${database_name.var};
SET database_name.dummy= marketing;
SHOW TABLES in ${database_name.dummy};do not use quotesuse format that is variableName...
3 More Replies
- 3674 Views
- 7 replies
- 2 kudos
Hi,we are in process of moving our Datawarehouse from sql server to databricks. we are in process of testing our Dimension Product table which has identity column for referencing in fact table as surrogate key. In Databricks Apply changes SCD type 2 ...
- 3674 Views
- 7 replies
- 2 kudos
Latest Reply
Hey. Yep, xxhash64 (or even just hash) generate numerical values for you. Combine with abs function to ensure the value is positive. In our team we used abs(hash()) ourselves... for maybe a day. Very quickly I observed a collision, and the data s...
6 More Replies
by
elgeo
• Valued Contributor II
- 28014 Views
- 9 replies
- 5 kudos
Hello. What would be the equivalent of the below in databricks?DECLARE @LastChangeDate as dateSET @LastChangeDate = GetDate()I already tried the below and worked. However I need to know how set a sql variable dynamicallySET da.dbname = test;SELECT "$...
- 28014 Views
- 9 replies
- 5 kudos
- 3455 Views
- 5 replies
- 4 kudos
I was trying to read some delta data from databricks[Hive metastore] sql endpoint using pyspark, but while doing so I encountered that all the values of the table after fetching are same as the column name.Even when I try to just show the data it giv...
- 3455 Views
- 5 replies
- 4 kudos
Latest Reply
Encountered the same issue and downgrading to 2.6.22 helped me resolve this issue.
4 More Replies
- 3307 Views
- 6 replies
- 5 kudos
We are using Databricks SQL Alerts to handle one scenario. We have written the queries for the same, also we have created the SQL Alert. However, I was looking for the best way to deploy it on Higher Environments like Pre-Production and Production.I ...
- 3307 Views
- 6 replies
- 5 kudos
- 9972 Views
- 6 replies
- 6 kudos
I need help or insight regarding the following errors. My instructors (Brooke Wenig with Conor Murphy) ran this code successfully on our course video, but I cannot replicate what she did. Here is the code and below it is the outcome from my Cours...
- 9972 Views
- 6 replies
- 6 kudos
Latest Reply
DELETE the original Parquet table as a separate statementHow can this be done? simple query "DROP TABLE .... " still failing with "cannot be found"
5 More Replies
- 3006 Views
- 6 replies
- 1 kudos
Hello everyone!So I want to ingest tables with schemas from the on-premise SQL server to Databricks Bronze layer with Delta Live Table and I want to do it using Azure Data Factory and I want the load to be a Snapshot batch load, not an incremental lo...
- 3006 Views
- 6 replies
- 1 kudos
Latest Reply
Hi @Parsa Bahraminejad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
5 More Replies
- 1161 Views
- 2 replies
- 1 kudos
Hii'm looking for performance test tool.I saw that there was apost about jmeter https://stackoverflow.com/questions/66913893/how-can-i-connect-jmeter-with-databricks-spark-cluster#comment118293766_66915965 , however, the jdbc paraeters are requesting...
- 1161 Views
- 2 replies
- 1 kudos
Latest Reply
Hi @Amit Cahanovich​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
1 More Replies