cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Joao_DE
by New Contributor III
  • 4237 Views
  • 3 replies
  • 1 kudos

Information about stoping JDBC connection

Hi everyone! I would like to know how spark stops the connection when reading from a sql database using the JDBC format. Also, if there is a way to check when the connection is active or manually stop it, I also would like to know. Thank you in advan...

  • 4237 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @João Peixoto​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
2 More Replies
SID
by New Contributor
  • 1995 Views
  • 2 replies
  • 0 kudos

Partner onboarding next steps

Hi team, our application for partner onboarding was processed a year ago. The last email we received was from Procurement Operations Specialist confirming our information has been added to the system. However, we never received any advise on the next...

  • 1995 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Siddhesh Gaikwad​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...

  • 0 kudos
1 More Replies
pankajBhatt
by New Contributor II
  • 2121 Views
  • 1 replies
  • 1 kudos

Databricks not able to access latest files in Azure ADLS Gen1

I have mounted my path from Databricks to AzureADLS Gen1. using SPN as service accuntuntill yesterday everything was ok, but today I see, I can view all older deleted folders. I can not see them in ADLS . but my databricks dbutils.fs.ls() shows them....

  • 2121 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @pankaj bhatt​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
744291
by New Contributor III
  • 2288 Views
  • 4 replies
  • 0 kudos

Anyone having extra voucher who are not plannning to give exams

Anyone having extra voucher who are not plannning to give exams

  • 2288 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Rituparna Das​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...

  • 0 kudos
3 More Replies
Trung
by Contributor
  • 2700 Views
  • 4 replies
  • 0 kudos

best practice for managing resources to avoid creators leaving out of the project

Dear DB community!As I know, when the resource creator is out of the project all resources they create will be deleted as well. So the question is that: can I assign the owner role to a group, that will help protect the resource from deletion or not...

  • 2700 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @trung nguyen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 0 kudos
3 More Replies
kll
by New Contributor III
  • 3734 Views
  • 2 replies
  • 0 kudos

Unable to render widget to display map within Jupyter notebook output cell

I am attempting to render a map within jupyter notebook and keep bumping into output limit. Below is my code: import pydeck as pdk   import pandas as pd       COLOR_BREWER_BLUE_SCALE = [   [240, 249, 232],   [204, 235, 197],   [168, 221, 181], ...

  • 3734 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Keval Shah​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...

  • 0 kudos
1 More Replies
KVNARK
by Honored Contributor II
  • 2747 Views
  • 2 replies
  • 6 kudos

Resolved! Microsoft ADB file from BLOB into Azure Databricks

Can anyone let me know how we can load the database file into Azure Databricks from the azure blob.

  • 2747 Views
  • 2 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @KVNARK .​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wil...

  • 6 kudos
1 More Replies
Michael_Papadop
by New Contributor II
  • 14361 Views
  • 3 replies
  • 1 kudos

How can I set the status of a databricks job as skipped via python?

I have a basic 2 task job. The 1st notebook (task) checks whether the source file has changes and if so then refreshes a corresponding materialized view. In case we have no changes then I use dbutils.jobs.taskValues.set(key = "skip_job", value = 1) &...

  • 14361 Views
  • 3 replies
  • 1 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@Michael Papadopoulos​ usually that should not be the case i think, as for task level we have 3 level notifications ( success, failure,start), where as whole job level skip option is available to discard notification . will see if some one from commu...

  • 1 kudos
2 More Replies
thushar
by Contributor
  • 10758 Views
  • 5 replies
  • 0 kudos

Optimize & Compaction

Hi,From which data bricks runtime will support Optimize and compaction

  • 10758 Views
  • 5 replies
  • 0 kudos
Latest Reply
Joe_Suarez
New Contributor III
  • 0 kudos

Optimize and compaction are operations commonly used in Apache Spark for optimizing and improving the performance of data storage and processing. Databricks, which is a cloud-based platform for Apache Spark, provides support for these operations on v...

  • 0 kudos
4 More Replies
youssefmrini
by Databricks Employee
  • 2120 Views
  • 1 replies
  • 0 kudos
  • 2120 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can ...

  • 0 kudos
youssefmrini
by Databricks Employee
  • 2761 Views
  • 1 replies
  • 0 kudos
  • 2761 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Feel free to read my medium blog where I summarized all the featureshttps://medium.com/@youssefmrini/databricks-workflows-features-oriented-3f9ec025301a

  • 0 kudos
beer
by New Contributor II
  • 2429 Views
  • 3 replies
  • 0 kudos

Didn't receive my Databricks Spark 3.0 certification

I took the exam yesterday and passed the test. I haven't received any email from Databricks Academy. How long would it take to receive the certification?

  • 2429 Views
  • 3 replies
  • 0 kudos
Latest Reply
beer
New Contributor II
  • 0 kudos

This is resolved.

  • 0 kudos
2 More Replies
Rik
by New Contributor III
  • 3587 Views
  • 2 replies
  • 0 kudos

Incorrect error when adding an IP access list

I have disabled the IP Access List on my workspace and am trying to add an IP list through the IP Access List API. However, when adding a list, I get the INVALID_STATE response.The docs mention this is because:"If the new list would block the calling...

  • 3587 Views
  • 2 replies
  • 0 kudos
Latest Reply
Rik
New Contributor III
  • 0 kudos

"One possible workaround could be to (1) temporarily enable the IP Access List feature, (2) add the necessary IP addresses to the list, and then (3) disable the feature again. This way, you can add the IP addresses you need without blocking the curre...

  • 0 kudos
1 More Replies
Indika_debnath
by New Contributor II
  • 6430 Views
  • 9 replies
  • 0 kudos

Databricks Certification voucher not received

Hello team,I have attended the webinar Databricks Certification Overview Series- Data Engineer on Jan 17Completed the Databricks Lakehouse fundamentals accreditation and Completed the survey.As per communication it is expected that I will receive Dat...

  • 6430 Views
  • 9 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Indika Debnath​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 0 kudos
8 More Replies
Labels