cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

MShee
by New Contributor II
  • 1181 Views
  • 1 replies
  • 1 kudos
  • 1181 Views
  • 1 replies
  • 1 kudos
Latest Reply
NandiniN
Honored Contributor
  • 1 kudos

Hello @M Shee​ ,In a drop down you can select a value from a list of provided values, not type the values in. What you might be interested in is a combobox - It is combination of text and dropdown. It allows to select a value from a provided list or ...

  • 1 kudos
Hubert-Dudek
by Esteemed Contributor III
  • 787 Views
  • 1 replies
  • 4 kudos

lnkd.in

Databricks has introduced a new feature that allows users to send SQL statements to their database via REST API. Users can easily integrate this feature with any tool by simply posting their queries to the /api/2.0/sql/statements/ endpoint. With this...

statmentapi
  • 787 Views
  • 1 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Hubert Dudek​, Your positive feedback and enthusiasm for our products mean a lot to us, and we appreciate your support.We take great pride in creating products that meet the needs of our customers, and your kind words validate our efforts. Your p...

  • 4 kudos
Lu_Wang_SA_DBX
by New Contributor III
  • 3518 Views
  • 1 replies
  • 3 kudos

We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave M...

We will host the first Databricks Bay Area User Group meeting in the Databricks Mountain View office on March 14 2:45-5:00 pm PT.We'll have Dave Mariani - CTO & Founder at AtScale, and Riley Phillips - Enterprise Solution Engineer at Matillion to sha...

David Mariana - CTO, AtScale Riley Phillips - Enterprise Solution Engineer, Matillion
  • 3518 Views
  • 1 replies
  • 3 kudos
Latest Reply
amitabharora
New Contributor II
  • 3 kudos

Looking forward.

  • 3 kudos
Everton_Costa
by New Contributor II
  • 1317 Views
  • 2 replies
  • 1 kudos
  • 1317 Views
  • 2 replies
  • 1 kudos
Latest Reply
Cami
Contributor III
  • 1 kudos

I hope it helps:SELECT DATEADD(DAY, rnk - 1, '{{StartDate}}') FROM ( WITH lv0(c) AS( SELECT 1 as c UNION ALL SELECT 1 ) , lv1 AS ( Select t1.c from lv0 t1 cross JOIN lv0 t2 ) , lv2 AS ( Select t1....

  • 1 kudos
1 More Replies
JacintoArias
by New Contributor III
  • 5150 Views
  • 6 replies
  • 2 kudos

Resolved! Spark predicate pushdown on parquet files when using limit

Hi,While developing an ETL for a large dataset I want to get a sample of the top rows to check that my the pipeline "just runs", so I add a limit clause when reading the dataset.I'm surprised to see that instead of creating a single task as in a sho...

  • 5150 Views
  • 6 replies
  • 2 kudos
Latest Reply
JacekLaskowski
New Contributor III
  • 2 kudos

It's been a while since the question was asked, and in the meantime Delta Lake 2.2.0 hit the shelves with the exact feature the OP asked about, i.e. LIMIT pushdown:LIMIT pushdown into Delta scan. Improve the performance of queries containing LIMIT cl...

  • 2 kudos
5 More Replies
rsamant07
by New Contributor III
  • 1518 Views
  • 3 replies
  • 2 kudos

Serverless SQL Cluster giving error with Powerbi

Power bu Giving this error while accessing delta table using serverless sql endpoint. Error while using path /mnt/xyz/_delta_log/00000000000000000000.checkpoint for resolving path '/xyz/_delta_log/00000000000000000000.checkpoint' within mount at '/mn...

  • 1518 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Rahul Samant​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 2 kudos
2 More Replies
maaaxx
by New Contributor III
  • 1276 Views
  • 3 replies
  • 4 kudos

A customized python library in cluster to access ADLS vis secret

Hello dear community,in our current project, we would like to develop a customized python library and deploy this library to all of the cluster to manage access control. You might ask why via a conventional way like external storage, well, we do not ...

  • 1276 Views
  • 3 replies
  • 4 kudos
Latest Reply
Anonymous
Not applicable
  • 4 kudos

Hi @Yuan Gao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we c...

  • 4 kudos
2 More Replies
andrcami1990
by New Contributor II
  • 4418 Views
  • 2 replies
  • 2 kudos

Resolved! Connect GraphQL to Data Bricks

Hi I am new to Databricks however I need to expose data found in the delta lake directly to GraphQL to be queried by several applications. Is there a connector or something similar to GraphQL that works with Databricks?

  • 4418 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Andrew Camilleri​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...

  • 2 kudos
1 More Replies
maxutil
by New Contributor II
  • 4358 Views
  • 2 replies
  • 3 kudos

Resolved! SQL select string and turn it into a decimal

select col as original, col::double as val_double, col::float as val_float, col::decimal(10,4) as val_decimal, to_number(col, '99999.99999') as val_tonum from int_fx_conversion_rate;The original value of col is a string such as '1...

  • 4358 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @Chris Chung​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...

  • 3 kudos
1 More Replies
JaiT
by New Contributor II
  • 1227 Views
  • 2 replies
  • 2 kudos

Resolved! DataBricks Workspace Environment

Hi, I am new to DataBricks and have started learning about it. I wanted to know if I can use the DataBricks workspace without the 3 Cloud Providers, i.e. AWS, Azure and GCP.If yes, then how?

  • 1227 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Jai Chitkara​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 2 kudos
1 More Replies
Alyayman
by Contributor
  • 1825 Views
  • 3 replies
  • 2 kudos
  • 1825 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Aly Ayman​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ...

  • 2 kudos
2 More Replies
Manuchito
by New Contributor
  • 1434 Views
  • 2 replies
  • 1 kudos

Resolved! Data Engineering with Databricks V2 not available in Partner

I cannot access the course anymore, it's shows it's under maintenance. For how long this will be? Is there any way to access it's videos for the Data Engineer Associate part?

  • 1434 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Juan Manuel Moviglia​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tel...

  • 1 kudos
1 More Replies
YSF
by New Contributor III
  • 9896 Views
  • 2 replies
  • 2 kudos

Resolved! How do I use the Python Logging Module in a Repo?

I have a repo that have python files that use the built in logging module. Additionally in some of the notebooks of the repo I want to use logging.debug()/logging.info() instead of print statements everywhere. However when I use the root logger or cr...

Screenshot 2023-02-28 143417
  • 9896 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Yusuf Khan​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...

  • 2 kudos
1 More Replies
uzairm
by New Contributor III
  • 5993 Views
  • 2 replies
  • 2 kudos

Resolved! ThreadPoolExecutor in Databricks

I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...

  • 5993 Views
  • 2 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @uzair mustafa​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...

  • 2 kudos
1 More Replies
haggholm
by New Contributor
  • 1877 Views
  • 2 replies
  • 1 kudos

Resolved! Query with ORDER BY fails with HiveThriftServerError "requirement failed: Subquery … has not finished"

Using ODBC or JDBC to read from a table fails when I attempt to use an ORDER BY clause. In one sample case, I have a fairly small table (just 1946 rows).select * from some_table order by some_fieldResult:java.lang.IllegalArgumentException: requiremen...

  • 1877 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @petter@hightouch.com Petter​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it doe...

  • 1 kudos
1 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels
Top Kudoed Authors