- 15996 Views
- 15 replies
- 12 kudos
Resolved! How to query delta lake using SQL desktop tools like SSMS or DBVisualizer
Is there a way to use sql desktop tools? because delta OSS or databricks does not provide desktop client (similar to azure data studio) to browse and query delta lake objects.I currently use databricks SQL , a webUI in the databricks workspace but se...
- 15996 Views
- 15 replies
- 12 kudos
- 12 kudos
DSR is Delta Standalone Reader. see more here - https://docs.delta.io/latest/delta-standalone.htmlIts a crate (and also now a py library) that allows you to connect to delta tables without using spark (e.g. directly from python and not using pyspa...
- 12 kudos
- 1862 Views
- 4 replies
- 1 kudos
Resolved! Control query caching using SQL statement execution API
I want to execute this statement using databricks SQL Statement Execution API. curl -X POST -H 'Authorization: Bearer <access-token>' -H 'Content-Type: application/json' -d '{"warehouse_id": "<warehouse_id>", "statement": "set us...
- 1862 Views
- 4 replies
- 1 kudos
- 12644 Views
- 13 replies
- 3 kudos
Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name
It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...
- 12644 Views
- 13 replies
- 3 kudos
- 4082 Views
- 5 replies
- 4 kudos
need help with Azure Databricks questions on CTE and SQL syntax within notebooks
Hi amazing community folks,Feel free to share your experience or knowledge regarding below questions:-1.) Can we pass a CTE sql statement into spark jdbc? i tried to do it i couldn't but i can pass normal sql (Select * from ) and it works. i heard th...
- 4082 Views
- 5 replies
- 4 kudos
- 4 kudos
CTE expressions are supported with the `prepareQuery` option. https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html A prefix that will form the final query together with query. As the specified query will be parenthesized as a subquery i...
- 4 kudos
- 2585 Views
- 6 replies
- 6 kudos
Why is writing to MSSQL Server 12.0 so slow directly from spark but nearly instant when I write to a csv and read it back
I have a dataframe that inexplicably takes forever to write to an MS SQL Server even though other dataframes, even much larger ones, write nearly instantly. I'm using this code:my_dataframe.write.format("jdbc") .option("url",sqlsUrl) .optio...
- 2585 Views
- 6 replies
- 6 kudos
- 6 kudos
Had a similar issue. I can do 1-4 million rows in 1 minute via SSIS ETL on SQL server. Table is 15 fields long. Looking at your code it seems you have many fields but nothing like 300-400 fields which can affect performance. You can check SQL Server ...
- 6 kudos
- 21974 Views
- 6 replies
- 3 kudos
Resolved! SQL Stored Procedure in Databricks
Hello. Is there an equivalent of SQL stored procedure in Databricks? Please note that I need a procedure that allows DML statements and not only Select statement as a function provides.Thank you in advance
- 21974 Views
- 6 replies
- 3 kudos
- 3 kudos
Is there a way to query Databricks using Power BI, via a stored procedure or similar? In my use case the query will have multiple parameters, that the user will provide values for at runtime. I don't think notebooks can be consumed directly from Powe...
- 3 kudos
- 2627 Views
- 5 replies
- 4 kudos
Resolved! Out of Memory/Connection Lost When Writing to External SQL Server from Databricks Using JDBC Connection
I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...
- 2627 Views
- 5 replies
- 4 kudos
- 5730 Views
- 6 replies
- 0 kudos
"Unable to upload to DBFS Query" Error running SQL warehouse query?
I have sql warehouse endpoints that work fine when querying from applications such as Tableau, but just running the included sample query against a running endpoint from the Query Editor from the workspace is returning "Unable to upload to DBFS Query...
- 5730 Views
- 6 replies
- 0 kudos
- 0 kudos
Hi @Marvin Ginns​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 0 kudos
- 8074 Views
- 8 replies
- 8 kudos
Resolved! Executing Notebooks - Run All Cells vs Run All Below
Due to dependencies, if one of our cells errors then we want the notebook to stop executing.We've noticed some odd behaviour when executing notebooks depending on if "Run all cells in this notebook" is selected from the header versus "Run All Below"....
- 8074 Views
- 8 replies
- 8 kudos
- 8 kudos
Has this been implemented? I have created a job using notebook. My notebook has 6 cells and if the code in first cell fails it should not run the rest of the cells
- 8 kudos
- 895 Views
- 1 replies
- 0 kudos
SQL function does not work in 'Create Function'
This SQL statement works fine by itself SELECT COUNT(1) FROM tablea f INNER JOIN tableb t ON lower(f.col1) = t.col1but if I want to use it inside a function:​CREATE OR REPLACE FUNCTION fn_abc(var1 ...
- 895 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Lily99 , I hope this message finds you well. Could you please try the code below and let me know the results? CREATE OR REPLACE FUNCTION fn_abc(var1 STRING, var2 STRING) RETURNS DOUBLECOMMENT 'test function'RETURN SELECT CASE WHEN EXISTS...
- 0 kudos
- 1170 Views
- 1 replies
- 0 kudos
DLT aggregation problem
I'm utilizing SQL to perform aggregation operations within a gold layer of a DLT pipeline. However, I'm encountering an error when running the pipeline while attempting to return a data frame using spark.sql.Could anyone please assist me with the SQL...
- 1170 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Yash_542965 , I hope this message finds you well. Could you please share a sample of code you are using so that we can check it further? Best regards,Lucas Rocha
- 0 kudos
- 5564 Views
- 11 replies
- 2 kudos
Sql Serverless Option is missing when using Azure Databricks Workspace with No Public IP and VNET Injection
HelloAfter creating an Databricks Workspace in Azure with No Public IP and VNET Injection, I'm unable to use DBSQL Serverless because the option to enable it in SQL warehouse Settings is missing. ​Is it by design? Is it a limitation when using Privat...
- 5564 Views
- 11 replies
- 2 kudos
- 2 kudos
Fixed, go to Profile -> Compute-> SQL Server Serverless -> On -> Save. For some reason this has been disabled for us.
- 2 kudos
- 30938 Views
- 11 replies
- 2 kudos
Resolved! Using current_timestamp as a default value in a delta table
I want to add a column to an existing delta table with a timestamp for when the data was inserted. I know I can do this by including current_timestamp with my SQL statement that inserts into the table. Is it possible to add a column to an existing de...
- 30938 Views
- 11 replies
- 2 kudos
- 2 kudos
Can you please provide information on the additional expenses related to using this feature compared to not utilizing it at all?
- 2 kudos
- 4486 Views
- 5 replies
- 1 kudos
Resolved! Can someone help me understand how compute pricing works.
Im looking at using Databricks internally for some Data Science projects. I am however very confused to how the pricing works and would like to obviously avoid high spending right now. Internal documentation and within Databricks All-Purpose Compute...
- 4486 Views
- 5 replies
- 1 kudos
- 1 kudos
Hello,I was able to get a very precise cost of Azure Databricks Clusters and Computers jobs, using the Microsoft API and Databricks APIThen I wrote a simple tool to extract and manipulate the API results and generate detailed cost reports that can be...
- 1 kudos
- 2038 Views
- 4 replies
- 3 kudos
Sending email alerts to non-databricks user
I am trying to send email alerts to a non databricks user. I am using Alerts feature available in SQL. Can someone help me with the steps.Do I first need to first add Notification Destination through Admin settings and then use this newly added desti...
- 2038 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Mitali Lad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 3 kudos