cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mado
by Valued Contributor II
  • 15238 Views
  • 4 replies
  • 3 kudos

Resolved! How to set a variable and use it in a SQL query

I want to define a variable and use it in a query, like below: %sql   SET database_name = "marketing"; SHOW TABLES in '${database_name}';However, I get the following error:ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near ''''(line 1, pos...

  • 15238 Views
  • 4 replies
  • 3 kudos
Latest Reply
CJS
New Contributor II
  • 3 kudos

Another option is demonstrated by this example:%sql SET database_name.var = marketing; SHOW TABLES in ${database_name.var}; SET database_name.dummy= marketing; SHOW TABLES in ${database_name.dummy};do not use quotesuse format that is variableName...

  • 3 kudos
3 More Replies
Priyag1
by Honored Contributor II
  • 1077 Views
  • 2 replies
  • 11 kudos

Query parameters in dashboardsQueries can optionally leverage parameters or static values. When a visualization based on a parameterized query is adde...

Query parameters in dashboardsQueries can optionally leverage parameters or static values. When a visualization based on a parameterized query is added to a dashboard, the visualization can either be configured to use a:Widget parameterWidget paramet...

  • 1077 Views
  • 2 replies
  • 11 kudos
Latest Reply
Natalie_NL
New Contributor II
  • 11 kudos

Hi, I build a dashboard with dashboard parameters, it works pretty easy!The advantage of dashboard parameters is that you do not have to set a default (it can be: all). This is convenient when you need to filter on values that change every time the q...

  • 11 kudos
1 More Replies
Chinu
by New Contributor III
  • 980 Views
  • 1 replies
  • 1 kudos

Resolved! How to create a raw data (with filter_by) to pull query history from now to 5 mins ago

Hi Team, Is it possible I can use "query_start_time_range" filter from the api call to get the query data only from now to 5 mins ago?Im using telegraf to call query history api but it looks like Im reaching the max return and I can't find how to use...

  • 980 Views
  • 1 replies
  • 1 kudos
Latest Reply
mathan_pillai
Valued Contributor
  • 1 kudos

Have you checked this https://docs.databricks.com/api-explorer/workspace/queryhistory/list you can list the queries based on time range as well. So you can try passing the fields in the filter_by parameter. Then pass the value as (current time - 5 m...

  • 1 kudos
Chinu
by New Contributor III
  • 900 Views
  • 1 replies
  • 1 kudos

Resolved! Databricks query history api with filter_by warehouse_id

Hi, Im trying to pull query history filtered by warehouse id but my url is not working. Do you have an example on how the url will looks like?I tried this --> https://**.cloud.databricks.com/api/2.0/sql/history/queries?filter_by={"warehouse_id":"193b...

  • 900 Views
  • 1 replies
  • 1 kudos
Latest Reply
Chinu
New Contributor III
  • 1 kudos

Oh, looks like I need to add this raw data. { "filter_by": {  "warehouse_ids": "193b15a590ed23d2" }}

  • 1 kudos
cmilligan
by Contributor II
  • 1430 Views
  • 1 replies
  • 0 kudos

Pull query that inserts into table

I'm trying to pull some data down for table history and am needing to view the query that inserted into a table. My team owns the process so I'm able to view the current query by just viewing it but I'm also wanting to capture changes over time witho...

  • 1430 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Coleman Milligan​ :Yes, in Databricks, you can use the built-in Delta Lake feature to track the history of changes made to a table, including the queries that inserted data into it.Here's an example of how to retrieve the queries that inserted data ...

  • 0 kudos
Julie1
by New Contributor II
  • 1793 Views
  • 2 replies
  • 1 kudos

Resolved! Query data not showing in custom alert notifications and QUERY_RESULT_ROWS

I've set up a custom alert notification for one of my Databricks SQL queries, and it triggers correctly, but I'm not able to get the actual results of the query to appear in the notification email. I've followed the example/template in the custom ale...

  • 1793 Views
  • 2 replies
  • 1 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 1 kudos

The actual query results are not displayed in the alert unfortunately. You can pass the alert condition etc, but not the raw results of the underlying query.I hope this will be added in the future.A workaround is to add a link to the query, so the r...

  • 1 kudos
1 More Replies
uzairm
by New Contributor III
  • 3053 Views
  • 2 replies
  • 1 kudos

My whole code is running on driver node, I want my code to run on worker nodes so that the memory of driver node is not exhausted. Please tell me improvement is my codes. My spark crashes frequently when the pulled data from s3 is huge.

I am running process which has 4 steps.Querying s3 file paths from dynamo DB based on certain parameters given by user. (function to do so provided by client, just have to import). Returns a list of filesCheck if those file paths have already been qu...

  • 3053 Views
  • 2 replies
  • 1 kudos
Latest Reply
Vartika
Moderator
  • 1 kudos

Hi @uzair mustafa​ Thank you for posting your question in our community! We are happy to assist you.Does @Suteja Kanuri​'s answer help? If it does, would you be happy to mark it as best?This will help other community members who may have similar ques...

  • 1 kudos
1 More Replies
ossinova
by Contributor II
  • 1030 Views
  • 2 replies
  • 2 kudos

PIVOT on month and quarter

I want to simplify this query:SELECT year(EntryDate) Year, AccountNumber, sum(CreditBase - DebitBase) FILTER(WHERE month(EntryDate) = 1) AS jan_total, sum(CreditBase - DebitBase) FILTER(WHERE month(EntryDate) = 2) AS feb_total, sum(CreditBase - Debi...

  • 1030 Views
  • 2 replies
  • 2 kudos
Latest Reply
Kaniz
Community Manager
  • 2 kudos

Hi @Oscar Dyremyhr​, We haven't heard from you since the last response from @Lakshay Goel​ ​, and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...

  • 2 kudos
1 More Replies
Dale_Ware
by New Contributor III
  • 1395 Views
  • 2 replies
  • 3 kudos

Resolved! How to query a table with backslashes in the name.

I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...

  • 1395 Views
  • 2 replies
  • 3 kudos
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 3 kudos

You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"

  • 3 kudos
1 More Replies
haggholm
by New Contributor
  • 1469 Views
  • 2 replies
  • 1 kudos

Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"

Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...

  • 1469 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @petter@hightouch.com Petter​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...

  • 1 kudos
1 More Replies
KVNARK
by Honored Contributor II
  • 625 Views
  • 1 replies
  • 4 kudos

Resolved! Query related to Storage account authentication

Use Case: Copy data from SharePoint List to Blob using Power AutomateShort Description:To Access the blob storage account from Power Automate. There are three authentication type:1. Access Key2. Service Principal3. Azure AD IntegratedWhich authentica...

  • 625 Views
  • 1 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

@KVNARK .​ :It's recommended to use the Azure AD Integrated authentication type. This authentication type allows you to use Azure Active Directory (AD) to authenticate and manage access to Blob Storage resources at the folder or container level using...

  • 4 kudos
Mado
by Valued Contributor II
  • 960 Views
  • 1 replies
  • 1 kudos

Resolved! How to query Databricks audit logs?

Hi,I would like to ask where the Databricks Audit Log files are stored on the DBFS.And is there any way that I can query log files?Thanks.

  • 960 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 1 kudos

Hi @Mohammad Saber​ ,I think first you need to configure audit log in databricks then you use it.Please refer below blog that will help you in this.Configure audit logging | Databricks on AWS

  • 1 kudos
desert_safari
by New Contributor II
  • 1275 Views
  • 2 replies
  • 0 kudos

Bugs with text query parameters?

Hi all,I have query with a handful of text query parameters, I need to use to insert data into a table from a dashboard e.g.INSERT INTO user_data (first_name, middle_name, last_name, city, country, zip_code) VALUES ('{{first_name}}', '{{middle_name}}...

  • 1275 Views
  • 2 replies
  • 0 kudos
Latest Reply
desert_safari
New Contributor II
  • 0 kudos

Sorry my client wants all 3 fields. The middle name has to be there. There are a few cases where people have just one name.

  • 0 kudos
1 More Replies
Labels