- 2774 Views
- 2 replies
- 1 kudos
- 2774 Views
- 2 replies
- 1 kudos
- 1 kudos
I hope it helps:SELECT DATEADD(DAY, rnk - 1, '{{StartDate}}') FROM ( WITH lv0(c) AS( SELECT 1 as c UNION ALL SELECT 1 ) , lv1 AS ( Select t1.c from lv0 t1 cross JOIN lv0 t2 ) , lv2 AS ( Select t1....
- 1 kudos
- 10427 Views
- 5 replies
- 1 kudos
Spark predicate pushdown on parquet files when using limit
Hi,While developing an ETL for a large dataset I want to get a sample of the top rows to check that my the pipeline "just runs", so I add a limit clause when reading the dataset.I'm surprised to see that instead of creating a single task as in a sho...
- 10427 Views
- 5 replies
- 1 kudos
- 1 kudos
It's been a while since the question was asked, and in the meantime Delta Lake 2.2.0 hit the shelves with the exact feature the OP asked about, i.e. LIMIT pushdown:LIMIT pushdown into Delta scan. Improve the performance of queries containing LIMIT cl...
- 1 kudos
- 3051 Views
- 3 replies
- 2 kudos
Serverless SQL Cluster giving error with Powerbi
Power bu Giving this error while accessing delta table using serverless sql endpoint. Error while using path /mnt/xyz/_delta_log/00000000000000000000.checkpoint for resolving path '/xyz/_delta_log/00000000000000000000.checkpoint' within mount at '/mn...
- 3051 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Rahul Samant​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
- 2 kudos
- 3815 Views
- 3 replies
- 4 kudos
A customized python library in cluster to access ADLS vis secret
Hello dear community,in our current project, we would like to develop a customized python library and deploy this library to all of the cluster to manage access control. You might ask why via a conventional way like external storage, well, we do not ...
- 3815 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Yuan Gao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we c...
- 4 kudos
- 10073 Views
- 2 replies
- 2 kudos
Resolved! Connect GraphQL to Data Bricks
Hi I am new to Databricks however I need to expose data found in the delta lake directly to GraphQL to be queried by several applications. Is there a connector or something similar to GraphQL that works with Databricks?
- 10073 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Andrew Camilleri​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 2 kudos
- 9153 Views
- 2 replies
- 3 kudos
Resolved! SQL select string and turn it into a decimal
select col as original, col::double as val_double, col::float as val_float, col::decimal(10,4) as val_decimal, to_number(col, '99999.99999') as val_tonum from int_fx_conversion_rate;The original value of col is a string such as '1...
- 9153 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Chris Chung​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...
- 3 kudos
- 3418 Views
- 2 replies
- 2 kudos
Resolved! DataBricks Workspace Environment
Hi, I am new to DataBricks and have started learning about it. I wanted to know if I can use the DataBricks workspace without the 3 Cloud Providers, i.e. AWS, Azure and GCP.If yes, then how?
- 3418 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jai Chitkara​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...
- 2 kudos
- 3131 Views
- 3 replies
- 2 kudos
- 3131 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Aly Ayman​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...
- 2 kudos
- 2557 Views
- 2 replies
- 1 kudos
Resolved! Data Engineering with Databricks V2 not available in Partner
I cannot access the course anymore, it's shows it's under maintenance. For how long this will be? Is there any way to access it's videos for the Data Engineer Associate part?
- 2557 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Juan Manuel Moviglia​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tel...
- 1 kudos
- 17754 Views
- 2 replies
- 4 kudos
Resolved! How do I use the Python Logging Module in a Repo?
I have a repo that have python files that use the built in logging module. Additionally in some of the notebooks of the repo I want to use logging.debug()/logging.info() instead of print statements everywhere. However when I use the root logger or cr...
- 17754 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Yusuf Khan​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...
- 4 kudos
- 19758 Views
- 2 replies
- 2 kudos
Resolved! ThreadPoolExecutor in Databricks
I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...
- 19758 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @uzair mustafa​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...
- 2 kudos
- 3685 Views
- 2 replies
- 1 kudos
Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"
Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...
- 3685 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @petter@hightouch.com Petter​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...
- 1 kudos
- 1641 Views
- 1 replies
- 5 kudos
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined...
Starting from #databricks runtime 12.2 LTS, implicit lateral column aliasing is now supported. This feature enables you to reuse an expression defined earlier in the same SELECT list, thus avoiding repetition of the same calculation.For instance, in ...
- 1641 Views
- 1 replies
- 5 kudos
- 5 kudos
Thanks for sharing this with the Databricks community.
- 5 kudos
- 2425 Views
- 2 replies
- 2 kudos
Resolved! User Memory problem on driver with DLT
Hi! I have a problem with user memory on driver (I have almost several mb of storage memory, 0 Execution memory and more than 7GB of JVM Memory on Heap in use). How it can be? I don't have any broadcast variables, joins or aggregations. All the pipe...
- 2425 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Yuliya Valava​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 2 kudos
- 2986 Views
- 4 replies
- 0 kudos
- 2986 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Youssef Mrini​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...
- 0 kudos
| User | Count |
|---|---|
| 1632 | |
| 790 | |
| 512 | |
| 349 | |
| 287 |
