cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SQL
by • New Contributor II
  • 2665 Views
  • 6 replies
  • 1 kudos

Presto hive table to delta table conversion

Hi Everyone, I am using the below sql query to generate the days in order in hive & it is working fine. The table got migrated to delta and my query is failing. It would be appreciated if someone helps me to figure out the issue.SQL Query :with  ex...

Data Engineering
Reply
  • 2665 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
thelogicplus
Contributor
  • 1 kudos

Hi @SQL @jose_gonzalez , Have you tried code conversion tool fromTravinto technologies  ? They have hive to delta table conversion 

  • 1 kudos
5 More Replies
CoopCoop
by • New Contributor III
  • 11074 Views
  • 6 replies
  • 7 kudos

Resolved! PDF Attachment on an Alert

Currently my Alert is an HTML table using data pointing to an SQL query.I was wondering if it is possible to attach the resulting table from this SQL query as a PDF to the alert email.If anyone has successfully implemented this, please let me know! T...

Data Engineering
Reply
  • 11074 Views
  • 6 replies
  • 7 kudos
View Replies
Latest Reply
Atanu
Databricks Employee
  • 7 kudos

Ok understood the concern, so basically the issue is with PDF rendering as much I understood. Let me know if I am wrong. Let me see if there is any improvement by our engineering team on this front.

  • 7 kudos
5 More Replies
AEM
by • New Contributor
  • 1379 Views
  • 0 replies
  • 0 kudos

How to set charset encoding in SQL view?

Hi! I have a SQL query that has a where-clause that checks a string attribute not being equal to e.g. 'シミュレータに接続されていません' (Japanese). This works fine when running the query in SQL Editor ad hoc, but creating a view with the same query, the special cha...

Data Engineering
Reply
  • 1379 Views
  • 0 replies
  • 0 kudos
Prasann_gupta
by • New Contributor
  • 10030 Views
  • 3 replies
  • 0 kudos

SQL CONTAINS Function is not working on Databricks

I am trying to use sql CONTAINS function in my sql query but it is throwing the below error :AnalysisException: Undefined function: 'CONTAINS'. This function is neither a registered temporary function nor a permanent function registered in the databa...

Data Engineering
Reply
  • 10030 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Prasann Gupta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
2 More Replies
oleole
by • Contributor
  • 8934 Views
  • 3 replies
  • 2 kudos

Resolved! Using "FOR XML PATH" in Spark SQL in sql syntax

I'm using spark version 3.2.1 on databricks (DBR 10.4 LTS), and I'm trying to convert sql server sql query to a new sql query that runs on a spark cluster using spark sql in sql syntax. However, spark sql does not seem to support XML PATH as a functi...

input output
Data Engineering
Reply
  • 8934 Views
  • 3 replies
  • 2 kudos
View Replies
Latest Reply
oleole
Contributor
  • 2 kudos

Posting the solution that I ended up using:%sql DROP TABLE if exists UserCountry; CREATE TABLE if not exists UserCountry ( UserID INT, Country VARCHAR(5000) ); INSERT INTO UserCountry SELECT L.UserID AS UserID, CONCAT_WS(',', co...

  • 2 kudos
2 More Replies
RichardSCarchit
by • New Contributor III
  • 34673 Views
  • 4 replies
  • 4 kudos

How to extract DDL from tables/views using a SQL query?

I want to write a SQL query that queries the information_schema to generate a list of objects, their columns, relationships etc. - basically a data dictionary. For each object I want to show the DDL code, and I know I can get it by executing show c...

Data Engineering
Reply
  • 34673 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Richard Architect​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sugg...

  • 4 kudos
3 More Replies
Dale_Ware
by • New Contributor III
  • 4071 Views
  • 2 replies
  • 3 kudos

Resolved! How to query a table with backslashes in the name.

I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...

Data Engineering
Reply
  • 4071 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"

  • 3 kudos
1 More Replies
cmilligan
by • Contributor II
  • 7756 Views
  • 6 replies
  • 1 kudos

Resolved! Reference a single item tuple using .format() in spark.sql()

I'm trying to pass the elements of a tuple into a sql query using .format(). This works fine when I have multiple items in my tuple, but when using a single item in a tuple I get an error.tuple1 = (1,2,3) tuple2 = (5,)   combo = tuple1 + tuple2   pri...

Result
Data Engineering
Reply
  • 7756 Views
  • 6 replies
  • 1 kudos
View Replies
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

Could you please post the code and the error that you are getting?

  • 1 kudos
5 More Replies
dulu
by • New Contributor III
  • 12497 Views
  • 5 replies
  • 6 kudos

split character string in cell with sql

I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...

hinh22 hinh223
Data Engineering
Reply
  • 12497 Views
  • 5 replies
  • 6 kudos
View Replies
Latest Reply
sher
Valued Contributor II
  • 6 kudos

you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows

  • 6 kudos
4 More Replies
naveen123
by • New Contributor II
  • 1830 Views
  • 3 replies
  • 3 kudos

Previous data is getting wiped off for delta tables

I am using only insert sql query to insert the hist. load but previous data getting deleted.Tried with python query also but same issue persists.Reading the data from gcp bucket(parquet file)writing the data into gcp bucket(delta file)..the deleted f...

Data Engineering
Reply
  • 1830 Views
  • 3 replies
  • 3 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 3 kudos

Share your query and also look for any error messages in the driver logs. This might help to undertand better what is happening.

  • 3 kudos
2 More Replies
Taha_Hussain
by • Databricks Employee
  • 7135 Views
  • 5 replies
  • 5 kudos

Connect a BI Tool: How do I access my lakehouse data from my BI tool?

You can find a rich ecosystem of tools that allow you to work with all your data in-place and deliver real-time business insights faster.This post will help you connect your existing tools like dbt, Fivetran, PowerBI, Tableau or SAP to ingest, transf...

Screen Shot 2022-08-09 at 11.54.56 PM
Data Engineering
Reply
  • 7135 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
Axserv
New Contributor II
  • 5 kudos

Hello Taha, here is a fairly recent video provided by Databricks on conncecting Power BI : Demo Video: Connect to Power BI Desktop from Databricks - YouTube

  • 5 kudos
4 More Replies
Swapnil1998
by • New Contributor III
  • 1047 Views
  • 0 replies
  • 0 kudos

How to query a MySQL Table from Databricks?

I wanted to query a MySQL Table using Databricks rather than reading the complete data using a dbtable option, which will help in incremental loads.remote_table = (spark.read .format("jdbc") .option("driver", driver) .option("url", URL) .option("quer...

Data Engineering
Reply
  • 1047 Views
  • 0 replies
  • 0 kudos
Anonymous
by • Not applicable
  • 12451 Views
  • 4 replies
  • 17 kudos

Resolved! SQL query takes too long to run

Hi all,I have a set up SQL query run with 5 hours but the SQL endpoint take too long to start up with each run. Currently I don't know how to fix this Could you please help me how to improve this?

Data Engineering
Reply
  • 12451 Views
  • 4 replies
  • 17 kudos
View Replies
Latest Reply
Unforgiven
Valued Contributor III
  • 17 kudos

or refer on this : Faster SQL Queries on Delta Lake with Dynamic File Pruninghttps://www.databricks.com/blog/2020/04/30/faster-sql-queries-on-delta-lake-with-dynamic-file-pruning.html

  • 17 kudos
3 More Replies
sage5616
by • Valued Contributor
  • 9680 Views
  • 11 replies
  • 10 kudos

Error in SQL statement: AnalysisException: Cannot up cast documents from array

Hi Everyone,I am getting the following error when running a SQL query and do not understand what it means or what can be done to resolve it. Any recommendations?View DDL:CREATE VIEW myschema.table ( accountId, agreementType, capture_file_name, ...

Data Engineering
Reply
  • 9680 Views
  • 11 replies
  • 10 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @Michael Okulik​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 10 kudos
10 More Replies
KrishZ
by • Contributor
  • 2316 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

Data Engineering
Reply
  • 2316 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
Labels