cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

CoopCoop
by New Contributor III
  • 8475 Views
  • 6 replies
  • 7 kudos

Resolved! PDF Attachment on an Alert

Currently my Alert is an HTML table using data pointing to an SQL query.I was wondering if it is possible to attach the resulting table from this SQL query as a PDF to the alert email.If anyone has successfully implemented this, please let me know! T...

  • 8475 Views
  • 6 replies
  • 7 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 7 kudos

Ok understood the concern, so basically the issue is with PDF rendering as much I understood. Let me know if I am wrong. Let me see if there is any improvement by our engineering team on this front.

  • 7 kudos
5 More Replies
AEM
by New Contributor
  • 845 Views
  • 0 replies
  • 0 kudos

How to set charset encoding in SQL view?

Hi! I have a SQL query that has a where-clause that checks a string attribute not being equal to e.g. 'シミュレータに接続されていません' (Japanese). This works fine when running the query in SQL Editor ad hoc, but creating a view with the same query, the special cha...

  • 845 Views
  • 0 replies
  • 0 kudos
Prasann_gupta
by New Contributor
  • 5455 Views
  • 3 replies
  • 0 kudos

SQL CONTAINS Function is not working on Databricks

I am trying to use sql CONTAINS function in my sql query but it is throwing the below error :AnalysisException: Undefined function: 'CONTAINS'. This function is neither a registered temporary function nor a permanent function registered in the databa...

  • 5455 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Prasann Gupta​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...

  • 0 kudos
2 More Replies
oleole
by Contributor
  • 5990 Views
  • 3 replies
  • 2 kudos

Resolved! Using "FOR XML PATH" in Spark SQL in sql syntax

I'm using spark version 3.2.1 on databricks (DBR 10.4 LTS), and I'm trying to convert sql server sql query to a new sql query that runs on a spark cluster using spark sql in sql syntax. However, spark sql does not seem to support XML PATH as a functi...

input output
  • 5990 Views
  • 3 replies
  • 2 kudos
Latest Reply
oleole
Contributor
  • 2 kudos

Posting the solution that I ended up using:%sql DROP TABLE if exists UserCountry; CREATE TABLE if not exists UserCountry ( UserID INT, Country VARCHAR(5000) ); INSERT INTO UserCountry SELECT L.UserID AS UserID, CONCAT_WS(',', co...

  • 2 kudos
2 More Replies
RichardSCarchit
by New Contributor III
  • 18363 Views
  • 4 replies
  • 4 kudos

How to extract DDL from tables/views using a SQL query?

I want to write a SQL query that queries the information_schema to generate a list of objects, their columns, relationships etc. - basically a data dictionary. For each object I want to show the DDL code, and I know I can get it by executing show c...

  • 18363 Views
  • 4 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Richard Architect​ I'm sorry you could not find a solution to your problem in the answers provided.Our community strives to provide helpful and accurate information, but sometimes an immediate solution may only be available for some issues.I sugg...

  • 4 kudos
3 More Replies
Dale_Ware
by New Contributor III
  • 1790 Views
  • 2 replies
  • 3 kudos

Resolved! How to query a table with backslashes in the name.

I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...

  • 1790 Views
  • 2 replies
  • 3 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"

  • 3 kudos
1 More Replies
cmilligan
by Contributor II
  • 5085 Views
  • 6 replies
  • 1 kudos

Resolved! Reference a single item tuple using .format() in spark.sql()

I'm trying to pass the elements of a tuple into a sql query using .format(). This works fine when I have multiple items in my tuple, but when using a single item in a tuple I get an error.tuple1 = (1,2,3) tuple2 = (5,)   combo = tuple1 + tuple2   pri...

Result
  • 5085 Views
  • 6 replies
  • 1 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 1 kudos

Could you please post the code and the error that you are getting?

  • 1 kudos
5 More Replies
dulu
by New Contributor III
  • 10116 Views
  • 6 replies
  • 6 kudos

split character string in cell with sql

I have the following input: I am looking for a way to split the characters in the item_order_detail column into 2 columns itemID and itemName. As below output table uses SQL function in databricks with spark_sql version 3.2.1.Can someone suggest a so...

hinh22 hinh223
  • 10116 Views
  • 6 replies
  • 6 kudos
Latest Reply
sher
Valued Contributor II
  • 6 kudos

you need to use explode functionhttps://stackoverflow.com/questions/61070630/spark-explode-column-with-json-array-to-rows

  • 6 kudos
5 More Replies
naveen123
by New Contributor II
  • 1265 Views
  • 3 replies
  • 3 kudos

Previous data is getting wiped off for delta tables

I am using only insert sql query to insert the hist. load but previous data getting deleted.Tried with python query also but same issue persists.Reading the data from gcp bucket(parquet file)writing the data into gcp bucket(delta file)..the deleted f...

  • 1265 Views
  • 3 replies
  • 3 kudos
Latest Reply
jose_gonzalez
Moderator
  • 3 kudos

Share your query and also look for any error messages in the driver logs. This might help to undertand better what is happening.

  • 3 kudos
2 More Replies
Taha_Hussain
by Valued Contributor II
  • 5503 Views
  • 5 replies
  • 5 kudos

Connect a BI Tool: How do I access my lakehouse data from my BI tool?

You can find a rich ecosystem of tools that allow you to work with all your data in-place and deliver real-time business insights faster.This post will help you connect your existing tools like dbt, Fivetran, PowerBI, Tableau or SAP to ingest, transf...

Screen Shot 2022-08-09 at 11.54.56 PM
  • 5503 Views
  • 5 replies
  • 5 kudos
Latest Reply
Axserv
New Contributor II
  • 5 kudos

Hello Taha, here is a fairly recent video provided by Databricks on conncecting Power BI : Demo Video: Connect to Power BI Desktop from Databricks - YouTube

  • 5 kudos
4 More Replies
Swapnil1998
by New Contributor III
  • 676 Views
  • 0 replies
  • 0 kudos

How to query a MySQL Table from Databricks?

I wanted to query a MySQL Table using Databricks rather than reading the complete data using a dbtable option, which will help in incremental loads.remote_table = (spark.read .format("jdbc") .option("driver", driver) .option("url", URL) .option("quer...

  • 676 Views
  • 0 replies
  • 0 kudos
Anonymous
by Not applicable
  • 7877 Views
  • 4 replies
  • 17 kudos

Resolved! SQL query takes too long to run

Hi all,I have a set up SQL query run with 5 hours but the SQL endpoint take too long to start up with each run. Currently I don't know how to fix this Could you please help me how to improve this?

  • 7877 Views
  • 4 replies
  • 17 kudos
Latest Reply
Unforgiven
Valued Contributor III
  • 17 kudos

or refer on this : Faster SQL Queries on Delta Lake with Dynamic File Pruninghttps://www.databricks.com/blog/2020/04/30/faster-sql-queries-on-delta-lake-with-dynamic-file-pruning.html

  • 17 kudos
3 More Replies
sage5616
by Valued Contributor
  • 6046 Views
  • 12 replies
  • 10 kudos

Error in SQL statement: AnalysisException: Cannot up cast documents from array

Hi Everyone,I am getting the following error when running a SQL query and do not understand what it means or what can be done to resolve it. Any recommendations?View DDL:CREATE VIEW myschema.table ( accountId, agreementType, capture_file_name, ...

  • 6046 Views
  • 12 replies
  • 10 kudos
Latest Reply
Anonymous
Not applicable
  • 10 kudos

Hi @Michael Okulik​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 10 kudos
11 More Replies
KrishZ
by Contributor
  • 1522 Views
  • 4 replies
  • 0 kudos

How to prevent sql queries in 2 notebooks from reading the same row from a Table ?

I have an SQL query to select and update rows in a table. I do this in batches of 300 rows (select 300 , update the selected 300 , select new 300 and update the newly selected and so on..) I run this query in 2 different notebooks concurrently to spe...

  • 1522 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krishna Zanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
3 More Replies
ejloh
by New Contributor II
  • 2240 Views
  • 3 replies
  • 1 kudos

SQL query with leads and lags

I'm trying to create a new column that fills in the nulls below. I tried using leads and lags but isn't turning out right. Basically trying to figure out who is in "possession" of the record, given the TransferFrom and TransferTo columns and sequence...

image image
  • 2240 Views
  • 3 replies
  • 1 kudos
Latest Reply
Vidula
Honored Contributor
  • 1 kudos

Hi there @Eric Lohbeck​ Does @Hubert Dudek​  response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!

  • 1 kudos
2 More Replies
Labels