- 21376 Views
- 11 replies
- 2 kudos
Resolved! Performance enhancement while writing dataframes into Parquet tables
Hi,I am trying to write the contents of a dataframe into a parquet table using the command below.df.write.mode("overwrite").format("parquet").saveAsTable("sample_parquet_table")The dataframe contains an extract from one of our source systems, which h...
- 21376 Views
- 11 replies
- 2 kudos
- 2 kudos
Writing an effective essay takes more than putting ideas on paper; it requires solid research, logical structure, clear arguments, and careful proofreading. Students working on challenging assignments may find essay writing help useful for organizing...
- 2 kudos
- 20183 Views
- 2 replies
- 2 kudos
ADD COLUMN IF NOT EXISTS does not recognize "IF NOT EXIST". How do I add a column to an existing delta table with SQL if the column does not already exist?
How do I add a column to an existing delta table with SQL if the column does not already exist?I am using the following code: <%sqlALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name type; >but it prints the error: <[PARSE_SYNTAX_ERROR] Synta...
- 20183 Views
- 2 replies
- 2 kudos
- 2 kudos
Here some SQL devilry that does not require SQL Scripting nor pythons. It adds a my_extra_column FLOAT to the target table with null values. I'm as shocked as anyone that this works, (and if it's undocumented behavior, someone in Databricks Command ...
- 2 kudos
- 7916 Views
- 5 replies
- 1 kudos
SQL week format issue its not showing result as 01(ww)
Hi Folks,I've requirement to show the week number as ww format. Please see the below codeselect weekofyear(date_add(to_date(current_date, 'yyyyMMdd'), +35)). also plz refre the screen shot for result.
- 7916 Views
- 5 replies
- 1 kudos
- 1 kudos
Great question! I've encountered similar week formatting challenges in SQL before. The weekofyear() function can sometimes behave unexpectedly depending on your database system and locale settings. I'd recommend checking if your system uses ISO week ...
- 1 kudos
- 42811 Views
- 1 replies
- 2 kudos
Requesting support for "SELECT TOP n from Table"
In notebook, It looks like if I need to select top N rows, I can rely on "LIMIT" keyword. It would be nice if you can support "TOP" as well The current approach to select 10 rows: select * from table1 LIMIT 10 Requesting TOP support: SELECT TOP 10 *...
- 42811 Views
- 1 replies
- 2 kudos
- 2 kudos
@SethuSrinivasan , Looks like this one slipped through the cracks — apologies if you've long since moved on, but posting anyway in case it helps someone hitting the same wall. In Databricks SQL, SELECT TOP n doesn't exist. You get the same result wit...
- 2 kudos
- 8103 Views
- 5 replies
- 1 kudos
sql delete?
<pre> Hello databricks people, I started working with databricks today. I have a sql script which I developed with sqlite3 on a laptop. I want to port the script to databricks. I started with two sql statements: select count(prop_id) from prop0; del...
- 8103 Views
- 5 replies
- 1 kudos
- 1 kudos
You aren't doing anything wrong logically, but Databricks requires row-level changes to happen on Delta Lake tables. Standard Spark tables (like those backed by raw Parquet) are often immutable. Have a look at the Language Manual for DELETE to ensure...
- 1 kudos
- 47929 Views
- 15 replies
- 6 kudos
Error writing parquet files
Hi, we are having this chain of errors every day in different files and processes:An error occurred while calling o11255.parquet.: org.apache.spark.SparkException: Job aborted.Caused by: org.apache.spark.SparkException: Job aborted due to stage failu...
- 47929 Views
- 15 replies
- 6 kudos
- 7572 Views
- 3 replies
- 2 kudos
SQL While do loops
Hello. Could you please suggest a workaround for a while do loop in Databricks SQL?WHILE LSTART>0 DO SET LSTRING=CONCAT(LSTRING, VSTRING2)Thank you in advance
- 7572 Views
- 3 replies
- 2 kudos
- 2 kudos
@elgeo Here are two alternatives.1. Use a recursive CTEWITH RECURSIVE loop_cte (lstart, lstring) AS ( SELECT 5 AS lstart, '' AS lstring UNION ALL SELECT lstart - 1, CONCAT(lstring, 'VSTRING2') FROM loop_cte WHERE lstart > 1 ) SELECT * FROM ...
- 2 kudos
- 44948 Views
- 13 replies
- 6 kudos
SQL Stored Procedure in Databricks
Hello. Is there an equivalent of SQL stored procedure in Databricks? Please note that I need a procedure that allows DML statements and not only Select statement as a function provides.Thank you in advance
- 44948 Views
- 13 replies
- 6 kudos
- 6 kudos
how to connect the stored procedures to power bi report builder, seems like it is not working
- 6 kudos
- 18621 Views
- 4 replies
- 4 kudos
Resolved! UUID/GUID Datatype in Databricks SQL
Hi all,I am trying to create a table with a GUID column.I have tried using GUID, UUID; but both of them are not working.Can someone help me with the syntax for adding a GUID column?Thanks!
- 18621 Views
- 4 replies
- 4 kudos
- 4 kudos
> What ANSI SQL data structure to use for UUID or GUID?I had similar question. The answer was `STRING`.
- 4 kudos
- 3986 Views
- 4 replies
- 2 kudos
How to search an array of words in a text field
Example:TABLE 1FIELD_TEXTI like salty food and Italian foodI have Italian foodbread, rice and beansmexican foodscoke, spritearray['italia', 'mex','coke']match TABLE1 X ARRAYResults:I like salty food and Italian foodI have Italian foodmexican foodsis ...
- 3986 Views
- 4 replies
- 2 kudos
- 2 kudos
Yes, it’s possible to search an array of words in a text field using SQL with LIKE clauses or regex functions, while PySpark provides higher scalability with functions like rlike and array_contains (Wikipedia explains that SQL is a domain-specific la...
- 2 kudos
- 27565 Views
- 14 replies
- 10 kudos
How to overcome missing query parameters in Databricks SQL?
Hi, there! I'm trying to build up my first dashboard based on Dataabricks SQL. As far as I can see if you define a query parameter you can't skip it further. I'm looking for any option where I can make my parameter optional. For instance, I have a ta...
- 27565 Views
- 14 replies
- 10 kudos
- 10 kudos
Hello guys, I'm not sure if you could solve this issue but here is how I've handled it:SELECT *FROM my_tableWHERE (CASE WHEN LEN(:my_parameter) > 0 THEN my_column = :my_parameter ELSE my_column = my_column END)I hope this can help!
- 10 kudos
- 34664 Views
- 16 replies
- 5 kudos
Resolved! Databricks SQL - Unable to Escape Dollar Sign ($) in Column Name
It seems that due to how Databricks processes SQL cells, it's impossible to escape the $ when it comes to a column name.I would expect the following to work:%sql SELECT 'hi' `$id`The backticks ought to escape everything. And indeed that's exactly wha...
- 34664 Views
- 16 replies
- 5 kudos
- 5 kudos
I needed a $ character in the context of Java format strings in the SQL printf function.With some trial and error, I found workarounds:A trailing backslash escapes the dollar character successfully, like soSELECT printf('%1$\s, %1$\s', 'hi') AS testC...
- 5 kudos
- 35128 Views
- 3 replies
- 0 kudos
spark is case sensitive? Spark is not case sensitive by default. If you have same column name in different case (Name, name), if you try to select eit...
spark is case sensitive?Spark is not case sensitive by default. If you have same column name in different case (Name, name), if you try to select either "Name" or "name" column you will get column ambiguity error.There is a way to handle this issue b...
- 35128 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi I had similar issues with parquet files when trying to query athena, fix was i had to inspect the parquet file since it contained columns such as "Name", "name" which the aws crawler / athena would interpret as a duplicate column since it would se...
- 0 kudos
- 5723 Views
- 4 replies
- 2 kudos
Error in SQL Warehouse: User is not part of org
I tried to start the Databricks SQL Warehouse cluster today, but received the following error message:Clusters are failing to launch. Cluster launch will be retired Request to create a cluster failed with an exception: PERMISSION_DENIED: User xxxx is...
- 5723 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi All,What you need to do is set a new owner. You can do this by clicking on permissions, then the setup icon, and choosing Assign new owner.It works for me.Thanks
- 2 kudos
- 9807 Views
- 6 replies
- 0 kudos
Error in SQL statement: UnsupportedOperationException: Cannot perform Merge as multiple source rows matched and attempted to modify the same
My code:- CREATE OR REPLACE TEMPORARY VIEW preprocessed_source ASSELECT Key_ID, Distributor_ID, Customer_ID, Customer_Name, ChannelFROM integr_masterdata.Customer_Master;-- Step 2: Perform the merge operation using the preprocessed source tableM...
- 9807 Views
- 6 replies
- 0 kudos
- 0 kudos
This error occurs; when we try to update all the cells of target_data without a single updated record in source_data(updates_data) , to resolve this issue add a update_time column with unix timestamp (or) make changes in at least one cell of streamin...
- 0 kudos