- 5524 Views
- 2 replies
- 3 kudos
Resolved! How to query a table with backslashes in the name.
I am trying to query a snowflake table from a databricks data frame similar to the following example.sql_query = "select * from Database.Schema.Table_/Name_/V"sqlContext.sql(f"{sql_query}" ) And I get an error like this.ParseException: [PARSE_SYNTAX_...
- 5524 Views
- 2 replies
- 3 kudos
- 3 kudos
You can use Double Quotes to get the plan. Using quotes it is important to write the table names in capital letters.SELECT * FROM "/TABLE/NAME"
- 3 kudos
- 21444 Views
- 5 replies
- 8 kudos
Resolved! How to get all the tables name with a specific column or columns in a database?
let's say there is a database db in which 700 tables are there, and we need to find all the tables name in which column "project_id" is present.just an example for ubderstanding the questions.
- 21444 Views
- 5 replies
- 8 kudos
- 8 kudos
databaseName = "db" desiredColumn = "project_id" database = spark.sql(f"show tables in {databaseName} ").collect() tablenames = [] for row in database: cols = spark.table(row.tableName).columns if desiredColumn in cols: tablenames.append(row....
- 8 kudos
- 3645 Views
- 3 replies
- 1 kudos
Resolved! Upsert When the Origin NOT Exists, but you need to change status in the target
Hi guys,I have a question about upsert/merge ... What do you do when que origin NOT exists, but you need to change status in the target​For exemple:01/03 : source dataset [ id =1 and status = Active] ; target table [*not exists*] >> in this time the ...
- 3645 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @William Scardua​ , Just adding to what @Vigneshraja Palaniraj​ replied.Reference: https://docs.databricks.com/sql/language-manual/delta-merge-into.htmlThanks & Regards,Nandini
- 1 kudos
- 6587 Views
- 5 replies
- 3 kudos
Resolved! Filter only Delta tables from an S3 folders list
Hello everyone,From a list of folders on s3, how can I filter which ones are Delta tables, without trying to read each one at a time?Thanks,Ovi
- 6587 Views
- 5 replies
- 3 kudos
- 3 kudos
Hello @Ovidiu Eremia​ ,To filter which folders on S3 contain Delta tables, you can look for the specific files that are associated with Delta tables. Delta Lake stores its metadata in a hidden folder named _delta_log, which is located at the root of ...
- 3 kudos
- 3926 Views
- 1 replies
- 1 kudos
Surrogate key using identity column.
I want to create a surrogate in the delta table And i used the identity column id-Generated as DefaultCan i insert rows into the delta table using only spark.sql like Insert query ? or i can also use write delta format options? If i use the df.write ...
- 3926 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Menaka Murugesan​ ,If you are using the identity column, I believe you would have created the table as below, (starts with value 1 and step 1)CREATE TABLE my_table ( id INT IDENTITY (1, 1) PRIMARY KEY, value STRING )You can insert values i...
- 1 kudos
- 10858 Views
- 3 replies
- 5 kudos
Resolved! PySpark UDF is taking long to process
Hi,I have UDF which runs for each spark dataframe row, does some complex processing and return string output. But it takes very long if data is 15000 rows. I have configured cluster with autoscaling, but its not spinning more servers.Please suggest h...
- 10858 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Sanjay Jain​ , Python UDFs are generally slower to process because it runs mostly in the driver which can also lead to OOM errors on Driver. To resolve this issue, please consider the below:Use spark built-in functions to do the same functionalit...
- 5 kudos
- 2724 Views
- 1 replies
- 1 kudos
- 2724 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @M Shee​ ,In a drop down you can select a value from a list of provided values, not type the values in. What you might be interested in is a combobox - It is combination of text and dropdown. It allows to select a value from a provided list or ...
- 1 kudos
- 8466 Views
- 1 replies
- 3 kudos
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave M...
We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to sha...
- 8466 Views
- 1 replies
- 3 kudos
- 3 kudos
Looking forward.
- 3 kudos
- 2749 Views
- 2 replies
- 1 kudos
- 2749 Views
- 2 replies
- 1 kudos
- 1 kudos
I hope it helps:SELECT DATEADD(DAY, rnk - 1, '{{StartDate}}') FROM ( WITH lv0(c) AS( SELECT 1 as c UNION ALL SELECT 1 ) , lv1 AS ( Select t1.c from lv0 t1 cross JOIN lv0 t2 ) , lv2 AS ( Select t1....
- 1 kudos
- 10299 Views
- 5 replies
- 1 kudos
Spark predicate pushdown on parquet files when using limit
Hi,While developing an ETL for a large dataset I want to get a sample of the top rows to check that my the pipeline "just runs", so I add a limit clause when reading the dataset.I'm surprised to see that instead of creating a single task as in a sho...
- 10299 Views
- 5 replies
- 1 kudos
- 1 kudos
It's been a while since the question was asked, and in the meantime Delta Lake 2.2.0 hit the shelves with the exact feature the OP asked about, i.e. LIMIT pushdown:LIMIT pushdown into Delta scan. Improve the performance of queries containing LIMIT cl...
- 1 kudos
- 2996 Views
- 3 replies
- 2 kudos
Serverless SQL Cluster giving error with Powerbi
Power bu Giving this error while accessing delta table using serverless sql endpoint. Error while using path /mnt/xyz/_delta_log/00000000000000000000.checkpoint for resolving path '/xyz/_delta_log/00000000000000000000.checkpoint' within mount at '/mn...
- 2996 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Rahul Samant​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
- 2 kudos
- 3780 Views
- 3 replies
- 4 kudos
A customized python library in cluster to access ADLS vis secret
Hello dear community,in our current project, we would like to develop a customized python library and deploy this library to all of the cluster to manage access control. You might ask why via a conventional way like external storage, well, we do not ...
- 3780 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Yuan Gao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we c...
- 4 kudos
- 9973 Views
- 2 replies
- 2 kudos
Resolved! Connect GraphQL to Data Bricks
Hi I am new to Databricks however I need to expose data found in the delta lake directly to GraphQL to be queried by several applications. Is there a connector or something similar to GraphQL that works with Databricks?
- 9973 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Andrew Camilleri​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 2 kudos
- 8980 Views
- 2 replies
- 3 kudos
Resolved! SQL select string and turn it into a decimal
select col as original, col::double as val_double, col::float as val_float, col::decimal(10,4) as val_decimal, to_number(col, '99999.99999') as val_tonum from int_fx_conversion_rate;The original value of col is a string such as '1...
- 8980 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Chris Chung​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...
- 3 kudos
- 3393 Views
- 2 replies
- 2 kudos
Resolved! DataBricks Workspace Environment
Hi, I am new to DataBricks and have started learning about it. I wanted to know if I can use the DataBricks workspace without the 3 Cloud Providers, i.e. AWS, Azure and GCP.If yes, then how?
- 3393 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jai Chitkara​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
- 2 kudos
| User | Count |
|---|---|
| 1623 | |
| 790 | |
| 501 | |
| 349 | |
| 287 |
