- 16330 Views
- 15 replies
- 12 kudos
Resolved! How to query delta lake using SQL desktop tools like SSMS or DBVisualizer
Is there a way to use sql desktop tools? because delta OSS or databricks does not provide desktop client (similar to azure data studio) to browse and query delta lake objects.I currently use databricks SQL , a webUI in the databricks workspace but se...
- 16330 Views
- 15 replies
- 12 kudos
- 12 kudos
DSR is Delta Standalone Reader. see more here - https://docs.delta.io/latest/delta-standalone.htmlIts a crate (and also now a py library) that allows you to connect to delta tables without using spark (e.g. directly from python and not using pyspa...
- 12 kudos
- 25229 Views
- 4 replies
- 3 kudos
Resolved! How to set a variable and use it in a SQL query
I want to define a variable and use it in a query, like below: %sql SET database_name = "marketing"; SHOW TABLES in '${database_name}';However, I get the following error:ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near ''''(line 1, pos...
- 25229 Views
- 4 replies
- 3 kudos
- 3 kudos
Another option is demonstrated by this example:%sql SET database_name.var = marketing; SHOW TABLES in ${database_name.var}; SET database_name.dummy= marketing; SHOW TABLES in ${database_name.dummy};do not use quotesuse format that is variableName...
- 3 kudos
- 1844 Views
- 2 replies
- 11 kudos
Query parameters in dashboardsQueries can optionally leverage parameters or static values. When a visualization based on a parameterized query is adde...
Query parameters in dashboardsQueries can optionally leverage parameters or static values. When a visualization based on a parameterized query is added to a dashboard, the visualization can either be configured to use a:Widget parameterWidget paramet...
- 1844 Views
- 2 replies
- 11 kudos
- 11 kudos
Hi, I build a dashboard with dashboard parameters, it works pretty easy!The advantage of dashboard parameters is that you do not have to set a default (it can be: all). This is convenient when you need to filter on values that change every time the q...
- 11 kudos
- 7782 Views
- 11 replies
- 7 kudos
Resolved! Insert into query showing error
Im trying to insert data from table with 7 columns to table with 37 columns
- 7782 Views
- 11 replies
- 7 kudos
- 5400 Views
- 1 replies
- 1 kudos
Resolved! How to create a raw data (with filter_by) to pull query history from now to 5 mins ago
Hi Team, Is it possible I can use "query_start_time_range" filter from the api call to get the query data only from now to 5 mins ago?Im using telegraf to call query history api but it looks like Im reaching the max return and I can't find how to use...
- 5400 Views
- 1 replies
- 1 kudos
- 1 kudos
Have you checked this https://docs.databricks.com/api-explorer/workspace/queryhistory/list you can list the queries based on time range as well. So you can try passing the fields in the filter_by parameter. Then pass the value as (current time - 5 m...
- 1 kudos
- 656 Views
- 0 replies
- 0 kudos
Does Spark Structured streaming query require write-ahead logs for at least once delivery with ForeachBatch output sink?
Is it sufficient to use the checkpoint directory with write-ahead logs?BTW. I use Kafka connector to read data from EventHub
- 656 Views
- 0 replies
- 0 kudos
- 1444 Views
- 1 replies
- 1 kudos
Resolved! Databricks query history api with filter_by warehouse_id
Hi, Im trying to pull query history filtered by warehouse id but my url is not working. Do you have an example on how the url will looks like?I tried this --> https://**.cloud.databricks.com/api/2.0/sql/history/queries?filter_by={"warehouse_id":"193b...
- 1444 Views
- 1 replies
- 1 kudos
- 1 kudos
Oh, looks like I need to add this raw data. { "filter_by": { "warehouse_ids": "193b15a590ed23d2" }}
- 1 kudos
- 7599 Views
- 1 replies
- 0 kudos
Pull query that inserts into table
I'm trying to pull some data down for table history and am needing to view the query that inserted into a table. My team owns the process so I'm able to view the current query by just viewing it but I'm also wanting to capture changes over time witho...
- 7599 Views
- 1 replies
- 0 kudos
- 0 kudos
@Coleman Milligan :Yes, in Databricks, you can use the built-in Delta Lake feature to track the history of changes made to a table, including the queries that inserted data into it.Here's an example of how to retrieve the queries that inserted data ...
- 0 kudos
- 6626 Views
- 2 replies
- 1 kudos
Resolved! Query data not showing in custom alert notifications and QUERY_RESULT_ROWS
I've set up a custom alert notification for one of my Databricks SQL queries, and it triggers correctly, but I'm not able to get the actual results of the query to appear in the notification email. I've followed the example/template in the custom ale...
- 6626 Views
- 2 replies
- 1 kudos
- 1 kudos
The actual query results are not displayed in the alert unfortunately. You can pass the alert condition etc, but not the raw results of the underlying query.I hope this will be added in the future.A workaround is to add a link to the query, so the r...
- 1 kudos
- 4664 Views
- 2 replies
- 1 kudos
My whole code is running on driver node, I want my code to run on worker nodes so that the memory of driver node is not exhausted. Please tell me improvement is my codes. My spark crashes frequently when the pulled data from s3 is huge.
I am running process which has 4 steps.Querying s3 file paths from dynamo DB based on certain parameters given by user. (function to do so provided by client, just have to import). Returns a list of filesCheck if those file paths have already been qu...
- 4664 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @uzair mustafa Thank you for posting your question in our community! We are happy to assist you.Does @Suteja Kanuri's answer help? If it does, would you be happy to mark it as best?This will help other community members who may have similar ques...
- 1 kudos
- 1509 Views
- 2 replies
- 2 kudos
PIVOT on month and quarter
I want to simplify this query:SELECT year(EntryDate) Year, AccountNumber, sum(CreditBase - DebitBase) FILTER(WHERE month(EntryDate) = 1) AS jan_total, sum(CreditBase - DebitBase) FILTER(WHERE month(EntryDate) = 2) AS feb_total, sum(CreditBase - Debi...
- 1509 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Oscar Dyremyhr, We haven't heard from you since the last response from @Lakshay Goel , and I was checking back to see if his suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful to ...
- 2 kudos
- 2199 Views
- 2 replies
- 3 kudos
Resolved! How to query a table with backslashes in the name.
I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...
- 2199 Views
- 2 replies
- 3 kudos
- 3 kudos
You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"
- 3 kudos
- 2109 Views
- 2 replies
- 1 kudos
Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"
Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...
- 2109 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @petter@hightouch.com Petter Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...
- 1 kudos
- 956 Views
- 1 replies
- 4 kudos
Resolved! Query related to Storage account authentication
Use Case: Copy data from SharePoint List to Blob using Power AutomateShort Description:To Access the blob storage account from Power Automate. There are three authentication type:1. Access Key2. Service Principal3. Azure AD IntegratedWhich authentica...
- 956 Views
- 1 replies
- 4 kudos
- 4 kudos
@KVNARK . :It's recommended to use the Azure AD Integrated authentication type. This authentication type allows you to use Azure Active Directory (AD) to authenticate and manage access to Blob Storage resources at the folder or container level using...
- 4 kudos
- 1716 Views
- 1 replies
- 1 kudos
Resolved! How to query Databricks audit logs?
Hi,I would like to ask where the Databricks Audit Log files are stored on the DBFS.And is there any way that I can query log files?Thanks.
- 1716 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Mohammad Saber ,I think first you need to configure audit log in databricks then you use it.Please refer below blog that will help you in this.Configure audit logging | Databricks on AWS
- 1 kudos