- 5979 Views
- 8 replies
- 2 kudos
- 5979 Views
- 8 replies
- 2 kudos
- 2 kudos
I also tried to leverage this endpoint (inferred from devtools): https://<workspace_id>.cloud.databricks.com/sql/api/dashboards/import with the exported dashboard (the dbdash file) in the request payload. It returns a 200 but nothing happens. Maybe s...
- 2 kudos
- 6119 Views
- 6 replies
- 12 kudos
Resolved! SSRS (on-prem) on Databricks SQL
Has anybody succeeded in querying Databricks SQL with an on-prem SSRS (so an on-prem Report Server and Report Builder)?I manage to create a connection that works (according to the connection test anyway), but the moment I try to create a dataset on t...
- 6119 Views
- 6 replies
- 12 kudos
- 12 kudos
Hi @-werners- , I am able to build connection between Microsoft visual studio and data bricks using Simba Spark ODBC driver. I can query delta tables sitting in Databricks from Microsoft Visual studio (SSRS). However, when I am deploying the report t...
- 12 kudos
- 6825 Views
- 6 replies
- 5 kudos
Resolved! How to deploy Databricks SQL queries and SQL Alerts from lower environment to higher environment?
We are using Databricks SQL Alerts to handle one scenario. We have written the queries for the same, also we have created the SQL Alert. However, I was looking for the best way to deploy it on Higher Environments like Pre-Production and Production.I ...
- 6825 Views
- 6 replies
- 5 kudos
- 11172 Views
- 5 replies
- 6 kudos
Resolved! Window function using last/last_value with PARTITION BY/ORDER BY has unexpected results
Hi, I'm wondering if this is the expected behavior when using last or last_value in a window function? I've written a query like this:select col1, col2, last_value(col2) over (partition by col1 order by col2) as column2_last from values ...
- 11172 Views
- 5 replies
- 6 kudos
- 6 kudos
For those stumbling across this; it seems LAST_VALUE emulates the same functionality as it does in SQL Server which does not, in most people's minds, have a proper row/range frame for the window. You can adjust it with the below syntax.I understand l...
- 6 kudos
- 9299 Views
- 4 replies
- 3 kudos
Databricks CI/CD Azure DevOps
Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.
- 9299 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Divyansh Jain Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 3 kudos
- 1992 Views
- 3 replies
- 3 kudos
Resolved! Databricks performance related documentation/books
Hi,Do you know any good resources about Databricks performance improvements(like improving query performances, monitoring/resolving performance bottlenecks etc)?Thanks
- 1992 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Ömer Özsakarya We haven't heard from you since the last response from @Lakshay Goel , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...
- 3 kudos
- 24638 Views
- 8 replies
- 4 kudos
Resolved! To get Number of rows inserted after performing an Insert operation into a table
Consider we have two tables A & B.qry = """INSERT INTO Table ASelect * from Table B where Id is null """spark.sql(qry)I need to get the number of records inserted after running this in databricks.
- 24638 Views
- 8 replies
- 4 kudos
- 4 kudos
Almost same advice than Hubert, I use the history of the delta table :df_history.select(F.col('operationMetrics')).collect()[0].operationMetrics['numOutputRows']You can find also other 'operationMetrics' values, like 'numTargetRowsDeleted'.
- 4 kudos
- 11414 Views
- 8 replies
- 2 kudos
Resolved! AnalysisException when running SQL queries
When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...
- 11414 Views
- 8 replies
- 2 kudos
- 2 kudos
This is most likely a conflict in the lib code, you can uninstall some libs on your cluster and try to narrow it down to the problematic one.
- 2 kudos
- 2473 Views
- 2 replies
- 3 kudos
Apache Hudi Table creation using hudi maven library
I installed hudi maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 in Dbricks Runtime Ver : 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) with spark config :spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCat...
- 2473 Views
- 2 replies
- 3 kudos
- 3 kudos
@Shanmugavel Chandrakasu %sql create table hudi_cow_pt_tbl ( id bigint, name string, ts bigint, dt string, hh string ) using hudi tblproperties ( type = 'cow', primaryKey = 'id', preCombineField = 'ts' ) partitioned by (dt, hh) location '/mnt/data/h...
- 3 kudos
- 9296 Views
- 2 replies
- 1 kudos
Resolved! How to Pull Cached SQL Table into Python Dictionary?
Hello,I have been working on this issue as a proof of concept - it would be extremely helpful to iterate through tables via loops in a few scenarios. I have a simple three column dimension that I added to a cached table.cache lazy table hedis_cache s...
- 9296 Views
- 2 replies
- 1 kudos
- 1 kudos
Got it to work, thank you for the tip! I needed to convert the dataframe over to a pandas dataframehttps://www.geeksforgeeks.org/convert-pyspark-dataframe-to-dictionary-in-python/
- 1 kudos
- 780 Views
- 0 replies
- 0 kudos
docs.databricks.com
What Serverless features are you using on Databricks? I am curious to know.Is it Databricks SQL Serverless or Model Serving?Proceed here to Compare serverless compute to other Databricks architectureshttps://docs.databricks.com/serverless-compute/ind...
- 780 Views
- 0 replies
- 0 kudos
- 2593 Views
- 3 replies
- 3 kudos
Resolved! Databricks SQL CREATE TABLE ParseException
I am trying to use the following code to create a deltatable%sqlCREATE TABLE rectangles(a INT, b INT, area INT GENERATED ALWAYS AS IDENTITY (START WITH 1, STEP BY 1))I don't know why but I am always getting the ParseException error.I tried all other ...
- 2593 Views
- 3 replies
- 3 kudos
- 3 kudos
How can we use SQL for my website emiratesevisaonline.com backend date?
- 3 kudos
- 5836 Views
- 8 replies
- 7 kudos
Databricks sql not able to evaluate expression current_user
Hi,I have a table as below:create table default.test_user(ID bigint NOT NULL GENERATED BY DEFAULT AS IDENTITY (START WITH 1 INCREMENT BY 1),usr1 varchar(255) NOT NULL,ts1 timestamp NOT NULL,usr2 varchar(255) NOT NULL,ts2 timestamp NOT NULL) USING Del...
- 5836 Views
- 8 replies
- 7 kudos
- 7 kudos
Hi @Himanshu Agrawal Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...
- 7 kudos
- 4637 Views
- 3 replies
- 2 kudos
Need help skipping previously executed cells in a failed Databricks job calling a notebook with multiple SQL cells
In Azure databricks, I have a job that calls a notebook which has multiple cells with sql queries. In case of any cell fails and when we restart the databricks job then how to skip previous cell which already ran and start only from the failed cell? ...
- 4637 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Sandip Rath Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 2 kudos
- 9120 Views
- 1 replies
- 0 kudos
Read CSV file using SQL
Hi, I am trying to reverse engineer to get to the source file for a table. Looking at the query history, I came across SQL string which loads data from file to table, however the code looks little mystery to me. I haven't come across idbfs, Can someb...
- 9120 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, I am trying to reverse engineer to get to the source file for a table. Looking at the query history, I came across SQL string which loads data from file to table, however the code looks little mystery to me. I haven't come across idbfs, Can someb...
- 0 kudos