cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Lily99
by New Contributor
  • 634 Views
  • 1 replies
  • 0 kudos

SQL function does not work in 'Create Function'

This SQL statement works fine by itself SELECT COUNT(1) FROM tablea f INNER JOIN tableb t ON lower(f.col1) = t.col1but if I want to use it inside a function:​CREATE OR REPLACE FUNCTION fn_abc(var1 ...

  • 634 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
New Contributor III
  • 0 kudos

Hello @Lily99 , I hope this message finds you well. Could you please try the code below and let me know the results? CREATE OR REPLACE FUNCTION fn_abc(var1 STRING, var2 STRING) RETURNS DOUBLECOMMENT 'test function'RETURN SELECT    CASE    WHEN EXISTS...

  • 0 kudos
Yash_542965
by New Contributor II
  • 956 Views
  • 1 replies
  • 0 kudos

DLT aggregation problem

I'm utilizing SQL to perform aggregation operations within a gold layer of a DLT pipeline. However, I'm encountering an error when running the pipeline while attempting to return a data frame using spark.sql.Could anyone please assist me with the SQL...

  • 956 Views
  • 1 replies
  • 0 kudos
Latest Reply
lucasrocha
New Contributor III
  • 0 kudos

Hello @Yash_542965 , I hope this message finds you well. Could you please share a sample of code you are using so that we can check it further? Best regards,Lucas Rocha

  • 0 kudos
Anonymous
by Not applicable
  • 4085 Views
  • 11 replies
  • 2 kudos

Sql Serverless Option is missing when using Azure Databricks Workspace with No Public IP and VNET Injection

HelloAfter creating an Databricks Workspace in Azure with No Public IP and VNET Injection, I'm unable to use DBSQL Serverless because the option to enable it in SQL warehouse Settings is missing. ​Is it by design? Is it a limitation when using Privat...

  • 4085 Views
  • 11 replies
  • 2 kudos
Latest Reply
Armanillo
New Contributor II
  • 2 kudos

Fixed, go to Profile -> Compute->  SQL Server Serverless -> On -> Save. For some reason this has been disabled for us.

  • 2 kudos
10 More Replies
deng77
by New Contributor III
  • 21452 Views
  • 11 replies
  • 2 kudos

Resolved! Using current_timestamp as a default value in a delta table

I want to add a column to an existing delta table with a timestamp for when the data was inserted. I know I can do this by including current_timestamp with my SQL statement that inserts into the table. Is it possible to add a column to an existing de...

  • 21452 Views
  • 11 replies
  • 2 kudos
Latest Reply
Vaibhav1000
New Contributor II
  • 2 kudos

Can you please provide information on the additional expenses related to using this feature compared to not utilizing it at all?

  • 2 kudos
10 More Replies
ClaudeR
by New Contributor III
  • 3282 Views
  • 5 replies
  • 1 kudos

Resolved! Can someone help me understand how compute pricing works.

Im looking at using Databricks internally for some Data Science projects. I am however very confused to how the pricing works and would like to obviously avoid high spending right now. Internal documentation and within Databricks All-Purpose Compute...

  • 3282 Views
  • 5 replies
  • 1 kudos
Latest Reply
GuillermoM
New Contributor II
  • 1 kudos

Hello,I was able to get a very precise cost of Azure Databricks Clusters and Computers jobs, using the Microsoft API and Databricks APIThen I wrote a simple tool to extract and manipulate the API results and generate detailed cost reports that can be...

  • 1 kudos
4 More Replies
Mits
by New Contributor II
  • 1440 Views
  • 4 replies
  • 3 kudos

Sending email alerts to non-databricks user

I am trying to send email alerts to a non databricks user. I am using Alerts feature available in SQL. Can someone help me with the steps.Do I first need to first add Notification Destination through Admin settings and then use this newly added desti...

  • 1440 Views
  • 4 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Mitali Lad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 3 kudos
3 More Replies
Hardy
by New Contributor III
  • 3760 Views
  • 5 replies
  • 4 kudos

The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption

I am trying to connect to SQL through JDBC from databricks notebook. (Below is my notebook command)val df = spark.read.jdbc(jdbcUrl, "[MyTableName]", connectionProperties) println(df.schema)When I execute this command, with DBR 10.4 LTS it works fin...

  • 3760 Views
  • 5 replies
  • 4 kudos
Latest Reply
DBXC
Contributor
  • 4 kudos

Try to add the following parameters to your SQL connection string. It fixed my problem for 13.X and 12.X;trustServerCertificate=true;hostNameInCertificate=*.database.windows.net; 

  • 4 kudos
4 More Replies
sp1
by New Contributor II
  • 9697 Views
  • 7 replies
  • 4 kudos

Resolved! Pass date value as parameter in Databricks SQL notebook

I want to pass yesterday date (In the example 20230115*.csv) in the csv file. Don't know how to create parameter and use it here.CREATE OR REPLACE TEMPORARY VIEW abc_delivery_logUSING CSVOPTIONS ( header="true", delimiter=",", inferSchema="true", pat...

  • 9697 Views
  • 7 replies
  • 4 kudos
Latest Reply
Asifpanjwani
New Contributor II
  • 4 kudos

@Kaniz @sp1 @Chaitanya_Raju @daniel_sahal Hi Everyone,I need the same scenario on SQL code, because my DBR cluster not allowed me to run python codeError: Unsupported cell during execution. SQL warehouses only support executing SQL cells.I appreciate...

  • 4 kudos
6 More Replies
amitca71
by Contributor II
  • 4096 Views
  • 5 replies
  • 4 kudos

Resolved! exception when using java SQL client

Hi,I try to use java sql. i can see that the query on databricks is executed properly.However, on my client i get exception (see below).versions:jdk: jdk-20.0.1 (tryed also with version 16, same results)https://www.oracle.com/il-en/java/technologies/...

  • 4096 Views
  • 5 replies
  • 4 kudos
Latest Reply
xebia
New Contributor II
  • 4 kudos

I am using java 17 and getting the same error.

  • 4 kudos
4 More Replies
nyehia
by Contributor
  • 3243 Views
  • 9 replies
  • 0 kudos

Can not access a sql file from Notebook

Hey,I have a repo of notebooks and SQL files, the typical way is to update/create notebooks in the repo then push it and CICD pipeline deploys the notebooks to the Shared workspace.the issue is that I can access the SQL files in the Repo but can not ...

tempsnip
  • 3243 Views
  • 9 replies
  • 0 kudos
Latest Reply
ok_1
New Contributor II
  • 0 kudos

ok

  • 0 kudos
8 More Replies
juanc
by New Contributor II
  • 3208 Views
  • 9 replies
  • 2 kudos

Activate spark extensions on SQL Endpoints

It would be possible to activate a custom extensions like Sedona (https://sedona.apache.org/download/databricks/ ) in SQL Endopoints?Example error:java.lang.ClassNotFoundException: org.apache.spark.sql.sedona_sql.UDT.GeometryUDT at org.apache.spark....

  • 3208 Views
  • 9 replies
  • 2 kudos
Latest Reply
naveenanto
New Contributor III
  • 2 kudos

@Kaniz What is the right way to add custom spark extension to sql warehouse clusters?

  • 2 kudos
8 More Replies
BeginnerBob
by New Contributor III
  • 18733 Views
  • 6 replies
  • 3 kudos

Convert Date to YYYYMMDD in databricks sql

Hi,I have a date column in a delta table called ADate. I need this in the format YYYYMMDD.In TSQL this is easy. However, I can't seem to be able to do this without splitting the YEAR, MONTH and Day and concatenating them together.Any ideas?

  • 18733 Views
  • 6 replies
  • 3 kudos
Latest Reply
JayDoubleYou42
New Contributor II
  • 3 kudos

I'll share I'm having a variant of the same issue. I have a varchar field in the form YYYYMMDD which I'm trying to join to another varchar field from another table in the form of MM/DD/YYYY. Does anyone know of a way to do this in SPARK SQL without s...

  • 3 kudos
5 More Replies
VVM
by New Contributor III
  • 9855 Views
  • 12 replies
  • 3 kudos

Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name

It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...

  • 9855 Views
  • 12 replies
  • 3 kudos
Latest Reply
Casper-Bang
New Contributor II
  • 3 kudos

What is the status on this bug report? its been over a year now. 

  • 3 kudos
11 More Replies
Constantine
by Contributor III
  • 9514 Views
  • 3 replies
  • 6 kudos

Resolved! CREATE TEMP TABLE FROM CTE

I have written a CTE in Spark SQL WITH temp_data AS (   ......   )   CREATE VIEW AS temp_view FROM SELECT * FROM temp_view; I get a cryptic error. Is there a way to create a temp view from CTE using Spark SQL in databricks?

  • 9514 Views
  • 3 replies
  • 6 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 6 kudos

In the CTE you can't do a CREATE. It expects an expression in the form of expression_name [ ( column_name [ , ... ] ) ] [ AS ] ( query )where expression_name specifies a name for the common table expression.If you want to create a view from a CTE, y...

  • 6 kudos
2 More Replies
pranathisg97
by New Contributor III
  • 1317 Views
  • 3 replies
  • 1 kudos

Resolved! Control query caching using SQL statement execution API

I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...

image.png
  • 1317 Views
  • 3 replies
  • 1 kudos
Latest Reply
TimFrazer
New Contributor II
  • 1 kudos

Did you ever find a solution to this problem?

  • 1 kudos
2 More Replies
Labels