cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Geetha
by New Contributor
  • 4122 Views
  • 2 replies
  • 0 kudos

Connection to Databricks through KNIME

I want to make a connection to Databricks with KNIME. To do this I am using "Create Databricks Environment" node. I have made the following configuration:1. Installed Databricks Simba JDBC driver 2. Made the necessary configuration in Create Databric...

  • 4122 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Geethanjali Nataraj​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell...

  • 0 kudos
1 More Replies
Jfoxyyc
by Valued Contributor
  • 2151 Views
  • 3 replies
  • 0 kudos

Delta Live Table with Autoloader issue

Using autoloader, I'm reading daily data partitioned by well. The data has a specific schema, but if there's no value for a column it isn't present in the json. For a specific column on a specific table I'm getting an error like:​Cannot convert long ...

  • 2151 Views
  • 3 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Jordan Fox​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we...

  • 0 kudos
2 More Replies
Joao_DE
by New Contributor III
  • 4418 Views
  • 3 replies
  • 1 kudos

Information about stoping JDBC connection

Hi everyone! I would like to know how spark stops the connection when reading from a sql database using the JDBC format. Also, if there is a way to check when the connection is active or manually stop it, I also would like to know. Thank you in advan...

  • 4418 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @João Peixoto​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
2 More Replies
SID
by Databricks Partner
  • 2093 Views
  • 2 replies
  • 0 kudos

Partner onboarding next steps

Hi team, our application for partner onboarding was processed a year ago. The last email we received was from Procurement Operations Specialist confirming our information has been added to the system. However, we never received any advise on the next...

  • 2093 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Siddhesh Gaikwad​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us...

  • 0 kudos
1 More Replies
pankajBhatt
by New Contributor II
  • 2175 Views
  • 1 replies
  • 1 kudos

Databricks not able to access latest files in Azure ADLS Gen1

I have mounted my path from Databricks to AzureADLS Gen1. using SPN as service accuntuntill yesterday everything was ok, but today I see, I can view all older deleted folders. I can not see them in ADLS . but my databricks dbutils.fs.ls() shows them....

  • 2175 Views
  • 1 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @pankaj bhatt​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so ...

  • 1 kudos
744291
by Databricks Partner
  • 2371 Views
  • 4 replies
  • 0 kudos

Anyone having extra voucher who are not plannning to give exams

Anyone having extra voucher who are not plannning to give exams

  • 2371 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Rituparna Das​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so...

  • 0 kudos
3 More Replies
Trung
by Contributor
  • 2803 Views
  • 4 replies
  • 0 kudos

best practice for managing resources to avoid creators leaving out of the project

Dear DB community!As I know, when the resource creator is out of the project all resources they create will be deleted as well. So the question is that: can I assign the owner role to a group, that will help protect the resource from deletion or not...

  • 2803 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @trung nguyen​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback...

  • 0 kudos
3 More Replies
kll
by New Contributor III
  • 3942 Views
  • 2 replies
  • 0 kudos

Unable to render widget to display map within Jupyter notebook output cell

I am attempting to render a map within jupyter notebook and keep bumping into output limit. Below is my code: import pydeck as pdk   import pandas as pd       COLOR_BREWER_BLUE_SCALE = [   [240, 249, 232],   [204, 235, 197],   [168, 221, 181], ...

  • 3942 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Keval Shah​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...

  • 0 kudos
1 More Replies
KVNARK
by Honored Contributor II
  • 2878 Views
  • 2 replies
  • 6 kudos

Resolved! Microsoft ADB file from BLOB into Azure Databricks

Can anyone let me know how we can load the database file into Azure Databricks from the azure blob.

  • 2878 Views
  • 2 replies
  • 6 kudos
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @KVNARK .​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback wil...

  • 6 kudos
1 More Replies
Michael_Papadop
by New Contributor II
  • 14861 Views
  • 3 replies
  • 1 kudos

How can I set the status of a databricks job as skipped via python?

I have a basic 2 task job. The 1st notebook (task) checks whether the source file has changes and if so then refreshes a corresponding materialized view. In case we have no changes then I use dbutils.jobs.taskValues.set(key = "skip_job", value = 1) &...

  • 14861 Views
  • 3 replies
  • 1 kudos
Latest Reply
karthik_p
Databricks Partner
  • 1 kudos

@Michael Papadopoulos​ usually that should not be the case i think, as for task level we have 3 level notifications ( success, failure,start), where as whole job level skip option is available to discard notification . will see if some one from commu...

  • 1 kudos
2 More Replies
thushar
by Databricks Partner
  • 11117 Views
  • 5 replies
  • 0 kudos

Optimize & Compaction

Hi,From which data bricks runtime will support Optimize and compaction

  • 11117 Views
  • 5 replies
  • 0 kudos
Latest Reply
Joe_Suarez
New Contributor III
  • 0 kudos

Optimize and compaction are operations commonly used in Apache Spark for optimizing and improving the performance of data storage and processing. Databricks, which is a cloud-based platform for Apache Spark, provides support for these operations on v...

  • 0 kudos
4 More Replies
youssefmrini
by Databricks Employee
  • 2232 Views
  • 1 replies
  • 0 kudos
  • 2232 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Support for Jupyter notebooks (.ipynb files) is available in Repos. You can clone repositories with .ipynb notebooks, work in Databricks UI, and then commit and push as .ipynb notebooks. Metadata such as a notebook dashboard is preserved. Admins can ...

  • 0 kudos
youssefmrini
by Databricks Employee
  • 2843 Views
  • 1 replies
  • 0 kudos
  • 2843 Views
  • 1 replies
  • 0 kudos
Latest Reply
youssefmrini
Databricks Employee
  • 0 kudos

Feel free to read my medium blog where I summarized all the featureshttps://medium.com/@youssefmrini/databricks-workflows-features-oriented-3f9ec025301a

  • 0 kudos
beer
by New Contributor II
  • 2499 Views
  • 3 replies
  • 0 kudos

Didn't receive my Databricks Spark 3.0 certification

I took the exam yesterday and passed the test. I haven't received any email from Databricks Academy. How long would it take to receive the certification?

  • 2499 Views
  • 3 replies
  • 0 kudos
Latest Reply
beer
New Contributor II
  • 0 kudos

This is resolved.

  • 0 kudos
2 More Replies
Labels