- 15001 Views
- 5 replies
- 4 kudos
Resolved! Pass date value as parameter in Databricks SQL notebook
I want to pass yesterday date (In the example 20230115*.csv) in the csv file. Don't know how to create parameter and use it here.CREATE OR REPLACE TEMPORARY VIEW abc_delivery_logUSING CSVOPTIONS ( header="true", delimiter=",", inferSchema="true", pat...
- 15001 Views
- 5 replies
- 4 kudos
- 4 kudos
@Retired_mod @sp1 @Chaitanya_Raju @daniel_sahal Hi Everyone,I need the same scenario on SQL code, because my DBR cluster not allowed me to run python codeError: Unsupported cell during execution. SQL warehouses only support executing SQL cells.I appr...
- 4 kudos
- 19335 Views
- 5 replies
- 2 kudos
Is it possible to use multiprocessing or threads to submit multiple queries to a database from Databricks in parallel?
We are trying to improve our overall runtime by running queries in parallel using either multiprocessing or threads. What I am seeing though is that when the function that runs this code is run on a separate process it doesnt return a dataFrame with...
- 19335 Views
- 5 replies
- 2 kudos
- 2 kudos
Thanks for the links mentioned above. But both of them uses raw python to achieve parallelism. Does this mean Spark (read PySpark) does exactly provisions for parallel execution of functions or even notebooks ? We used a wrapper notebook with ThreadP...
- 2 kudos
- 2179 Views
- 2 replies
- 7 kudos
docs.databricks.com
Rename and drop columns with Delta Lake column mapping. Hi all,Now databricks started supporting column rename and drop.Column mapping requires the following Delta protocols:Reader version 2 or above.Writer version 5 or above.Blog URL##Available in D...
- 2179 Views
- 2 replies
- 7 kudos
- 7 kudos
Above mentioned feature is not working in the DLT pipeline. if the scrip has more than 4 columns
- 7 kudos
- 7445 Views
- 8 replies
- 2 kudos
- 7445 Views
- 8 replies
- 2 kudos
- 2 kudos
I also tried to leverage this endpoint (inferred from devtools): https://<workspace_id>.cloud.databricks.com/sql/api/dashboards/import with the exported dashboard (the dbdash file) in the request payload. It returns a 200 but nothing happens. Maybe s...
- 2 kudos
- 7293 Views
- 6 replies
- 12 kudos
Resolved! SSRS (on-prem) on Databricks SQL
Has anybody succeeded in querying Databricks SQL with an on-prem SSRS (so an on-prem Report Server and Report Builder)?I manage to create a connection that works (according to the connection test anyway), but the moment I try to create a dataset on t...
- 7293 Views
- 6 replies
- 12 kudos
- 12 kudos
Hi @-werners- , I am able to build connection between Microsoft visual studio and data bricks using Simba Spark ODBC driver. I can query delta tables sitting in Databricks from Microsoft Visual studio (SSRS). However, when I am deploying the report t...
- 12 kudos
- 8538 Views
- 6 replies
- 5 kudos
Resolved! How to deploy Databricks SQL queries and SQL Alerts from lower environment to higher environment?
We are using Databricks SQL Alerts to handle one scenario. We have written the queries for the same, also we have created the SQL Alert. However, I was looking for the best way to deploy it on Higher Environments like Pre-Production and Production.I ...
- 8538 Views
- 6 replies
- 5 kudos
- 14621 Views
- 5 replies
- 7 kudos
Resolved! Window function using last/last_value with PARTITION BY/ORDER BY has unexpected results
Hi, I'm wondering if this is the expected behavior when using last or last_value in a window function? I've written a query like this:select col1, col2, last_value(col2) over (partition by col1 order by col2) as column2_last from values ...
- 14621 Views
- 5 replies
- 7 kudos
- 7 kudos
For those stumbling across this; it seems LAST_VALUE emulates the same functionality as it does in SQL Server which does not, in most people's minds, have a proper row/range frame for the window. You can adjust it with the below syntax.I understand l...
- 7 kudos
- 15505 Views
- 4 replies
- 3 kudos
Databricks CI/CD Azure DevOps
Hi All. I have a scenario where there are few .sql scripts present in my repo. Is there any way we can execute those SQLs on Databricks via Azure DevOps CI/CD pipeline?Please help.
- 15505 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Divyansh Jain Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers...
- 3 kudos
- 2424 Views
- 3 replies
- 3 kudos
Resolved! Databricks performance related documentation/books
Hi,Do you know any good resources about Databricks performance improvements(like improving query performances, monitoring/resolving performance bottlenecks etc)?Thanks
- 2424 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Ömer Özsakarya We haven't heard from you since the last response from @Lakshay Goel , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...
- 3 kudos
- 30109 Views
- 8 replies
- 4 kudos
Resolved! To get Number of rows inserted after performing an Insert operation into a table
Consider we have two tables A & B.qry = """INSERT INTO Table ASelect * from Table B where Id is null """spark.sql(qry)I need to get the number of records inserted after running this in databricks.
- 30109 Views
- 8 replies
- 4 kudos
- 4 kudos
Almost same advice than Hubert, I use the history of the delta table :df_history.select(F.col('operationMetrics')).collect()[0].operationMetrics['numOutputRows']You can find also other 'operationMetrics' values, like 'numTargetRowsDeleted'.
- 4 kudos
- 20672 Views
- 8 replies
- 2 kudos
Resolved! AnalysisException when running SQL queries
When running some SQL queries using spark.sql(...), we sometimes get a variant of the following error:AnalysisException: Undefined function: current_timestamp. This function is neither a built-in/temporary function, nor a persistent function that is ...
- 20672 Views
- 8 replies
- 2 kudos
- 2 kudos
This is most likely a conflict in the lib code, you can uninstall some libs on your cluster and try to narrow it down to the problematic one.
- 2 kudos
- 3037 Views
- 2 replies
- 3 kudos
Apache Hudi Table creation using hudi maven library
I installed hudi maven library org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0 in Dbricks Runtime Ver : 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) with spark config :spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCat...
- 3037 Views
- 2 replies
- 3 kudos
- 3 kudos
@Shanmugavel Chandrakasu %sql create table hudi_cow_pt_tbl ( id bigint, name string, ts bigint, dt string, hh string ) using hudi tblproperties ( type = 'cow', primaryKey = 'id', preCombineField = 'ts' ) partitioned by (dt, hh) location '/mnt/data/h...
- 3 kudos
- 10411 Views
- 2 replies
- 1 kudos
Resolved! How to Pull Cached SQL Table into Python Dictionary?
Hello,I have been working on this issue as a proof of concept - it would be extremely helpful to iterate through tables via loops in a few scenarios. I have a simple three column dimension that I added to a cached table.cache lazy table hedis_cache s...
- 10411 Views
- 2 replies
- 1 kudos
- 1 kudos
Got it to work, thank you for the tip! I needed to convert the dataframe over to a pandas dataframehttps://www.geeksforgeeks.org/convert-pyspark-dataframe-to-dictionary-in-python/
- 1 kudos
- 950 Views
- 0 replies
- 0 kudos
docs.databricks.com
What Serverless features are you using on Databricks? I am curious to know.Is it Databricks SQL Serverless or Model Serving?Proceed here to Compare serverless compute to other Databricks architectureshttps://docs.databricks.com/serverless-compute/ind...
- 950 Views
- 0 replies
- 0 kudos
- 7883 Views
- 3 replies
- 3 kudos
Resolved! Databricks SQL CREATE TABLE ParseException
I am trying to use the following code to create a deltatable%sqlCREATE TABLE rectangles(a INT, b INT, area INT GENERATED ALWAYS AS IDENTITY (START WITH 1, STEP BY 1))I don't know why but I am always getting the ParseException error.I tried all other ...
- 7883 Views
- 3 replies
- 3 kudos
- 3 kudos
How can we use SQL for my website emiratesevisaonline.com backend date?
- 3 kudos