cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

oleole
by Contributor
  • 5121 Views
  • 3 replies
  • 2 kudos

Resolved! Using "FOR XML PATH" in Spark SQL in sql syntax

I'm using spark version 3.2.1 on databricks (DBR 10.4 LTS), and I'm trying to convert sql server sql query to a new sql query that runs on a spark cluster using spark sql in sql syntax. However, spark sql does not seem to support XML PATH as a functi...

input output
  • 5121 Views
  • 3 replies
  • 2 kudos
Latest Reply
oleole
Contributor
  • 2 kudos

Posting the solution that I ended up using:%sql DROP TABLE if exists UserCountry; CREATE TABLE if not exists UserCountry ( UserID INT, Country VARCHAR(5000) ); INSERT INTO UserCountry SELECT L.UserID AS UserID, CONCAT_WS(',', co...

  • 2 kudos
2 More Replies
prasadvaze
by Valued Contributor
  • 5265 Views
  • 3 replies
  • 0 kudos

Error loading MANAGED table in unity catalog delta lake on azure. Anyone seen this issue? "ErrorClass=INVALID_PARAMETER_VALUE] Input path <file system name>.dfs.core.windows.net overlaps with other external tables"

00007160: 2023-01-30T14:22:06 [TARGET_LOAD ]E: Failed (retcode -1) to execute statement: 'COPY INTO `e2underwriting_dbo`.`product` FROM(SELECT cast(_c0 as INT) as `ProductID`, _c1 as `ShortName`, cast(_c2 as INT) as `Status`, cast(_c3 as TIMESTA...

  • 5265 Views
  • 3 replies
  • 0 kudos
Latest Reply
prasadvaze
Valued Contributor
  • 0 kudos

we have solved this issue related to Qlik replicate copying data into delta table

  • 0 kudos
2 More Replies
prabhjot
by New Contributor III
  • 1837 Views
  • 4 replies
  • 2 kudos

Resolved! Data lineage graph is not working

Hi Team,The issue - Data lineage graph is not working (16-feb, 17-18 Feb) –  I created the below tables but when I click the lineage graph not able to see the upstream or downstream table .... the + sign goes away after a few sec but not able to clic...

  • 1837 Views
  • 4 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Prabhjot wadhwa​, Is it working fine now?

  • 2 kudos
3 More Replies
weldermartins
by Honored Contributor
  • 2803 Views
  • 2 replies
  • 1 kudos

Resolved! How to make spark-submit work on windows?

I have Jupyter Notebook installed on my machine working normally. I tested running a Spark application by running the spark-submit command and it returns the message that the file was not found. What do you need to do to make it work?Below is a file ...

image
  • 2803 Views
  • 2 replies
  • 1 kudos
Latest Reply
Debayan
Esteemed Contributor III
  • 1 kudos

Hi, yet this is not tested in my lab, but could you please check and confirm if this works: https://stackoverflow.com/questions/37861469/how-to-submit-spark-application-on-cmd

  • 1 kudos
1 More Replies
ta_db
by New Contributor
  • 1113 Views
  • 2 replies
  • 0 kudos

Databricks SQL Endpoint Failing to create an external table on a parquet file with Decimal or Timestamp datatype

I'm using the Databricks SQL Endpoint and I'm attempting to create an external table on top of an existing parquet file. I can do this so long as my table definition does not include a reference to a decimal or timestamp/date datatype.ex. This worksC...

  • 1113 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hey there @T A​ Hope everything is going great!Does @Kaniz Fatma​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly? If not, would you be happy to give us more info...

  • 0 kudos
1 More Replies
LightUp
by New Contributor III
  • 4112 Views
  • 2 replies
  • 4 kudos

Converting SQL Code to SQL Databricks

I am new to Databricks. Please excuse my ignorance. My requirement is to convert the SQL query below into Databricks SQL. The query comes from EventLog table and the output of the query goes into EventSummaryThese queries can be found hereCREATE TABL...

image
  • 4112 Views
  • 2 replies
  • 4 kudos
Latest Reply
LightUp
New Contributor III
  • 4 kudos

Thank you @Joseph Kambourakis​  The part that is not clear to me from the how to rework the part circled in the image above. Even this part of the code does not work in databricks:DATEADD(month, DATEDIFF(month, 0, DATEADD(month , 1 , EventStartDateTi...

  • 4 kudos
1 More Replies
Sudeshna
by New Contributor III
  • 1237 Views
  • 2 replies
  • 3 kudos

How can i pass one of the values from one function to another as an argument in Databricks SQL?

For eg - CREATE OR REPLACE TABLE table2(a INT, b INT);INSERT INTO table2 VALUES (100, 200);CREATE OR REPLACE FUNCTION func1() RETURNS TABLE(a INT, b INT) RETURN (SELECT a+b, a*b from table2);create or replace function calc(p DOUBLE) RETURNS TABLE(val...

image
  • 1237 Views
  • 2 replies
  • 3 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 3 kudos

Yes, it is possible, but with different logic. For scalar, so calc(a) in select calc(a) from func1(); it can only be a query as a table for a scalar is not allowed. So please try something like:CREATE OR REPLACE FUNCTION func_table() RETURNS TABLE(a ...

  • 3 kudos
1 More Replies
shan_chandra
by Esteemed Contributor
  • 2446 Views
  • 1 replies
  • 2 kudos

Resolved! java.lang.ArithmeticException: Casting XXXXXXXXXXX to int causes overflow

My job started failing with the below error when inserting rows into a delta table. ailing with the below error when inserting rows (timestamp) to a delta table, it was working well before.java.lang.ArithmeticException: Casting XXXXXXXXXXX to int cau...

  • 2446 Views
  • 1 replies
  • 2 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 2 kudos

This is because the Integer type represents 4-byte signed integer numbers. The range of numbers is from -2147483648 to 2147483647.Kindly use double as the data type to insert the "2147483648" value in the delta table.In the below example, The second ...

  • 2 kudos
cristianc
by Contributor
  • 2493 Views
  • 9 replies
  • 0 kudos

Resolved! Query AWS Redshift from Databricks SQL

Greetings,In the documentation for Databricks SQL it states that it supports JDBC connections, however when connecting to AWS Redshift via the built in PostgreSQL driver ("CREATE TABLE sample USING JDBC" and "jdbc://postgresql:/..." URI) I'm getting ...

  • 2493 Views
  • 9 replies
  • 0 kudos
Latest Reply
cristianc
Contributor
  • 0 kudos

@Bilal Aslam​ anytime! Is there a place where customers could follow the timeline when such features are introduced?

  • 0 kudos
8 More Replies
Labels