cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Hubert-Dudek
by Esteemed Contributor III
  • 1202 Views
  • 1 replies
  • 8 kudos

Databricks will release a new navigation experience to enhance user efficiency and convenience. The update includes a single, unified homepage with a ...

Databricks will release a new navigation experience to enhance user efficiency and convenience. The update includes a single, unified homepage with a list of recently viewed objects, a single search box, and common Lakehouse categories at the top of ...

nav-2-graphic
  • 1202 Views
  • 1 replies
  • 8 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 8 kudos

Thank you for sharing @Hubert Dudek​ 

  • 8 kudos
Tjadi
by New Contributor III
  • 2449 Views
  • 2 replies
  • 4 kudos

Specifying cluster on running a job

Hi,Let's say that I am starting jobs with different parameters at a certain time each day in the following manner:response = requests.post( "https://%s/api/2.0/jobs/run-now" % (DOMAIN), headers={"Authorization": "Bearer %s" % TOKEN}, json={ ...

  • 2449 Views
  • 2 replies
  • 4 kudos
Latest Reply
karthik_p
Esteemed Contributor
  • 4 kudos

@Tjadi Peeters​ You can select option Autoscaling/Enhanced Scaling in workflows which will scale based on workload

  • 4 kudos
1 More Replies
Aasu
by New Contributor II
  • 2589 Views
  • 3 replies
  • 1 kudos

In azure databricks custom alert i am able to preview but when it goes to mail data gets missing

In azure databricks custom alert i am able to preview but when it goes to mail data gets missing

  • 2589 Views
  • 3 replies
  • 1 kudos
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, what is the custom alert about? Also, could you please raise an Azure support case on the same to triage this further? Please tag @Debayan​ with your next response which will notify me. Thank you!

  • 1 kudos
2 More Replies
Eric-JoelBlanco
by New Contributor II
  • 2754 Views
  • 1 replies
  • 2 kudos

Execute code on Application End

Hello,I want to execute a custom code onApplicationEnd. Outisde Databricks, I have used the Spark Listener onApplicationEnd without problems. But it is not working on Databricks (I tried listener onJobEnd and this one worked).I have also tried Spark ...

  • 2754 Views
  • 1 replies
  • 2 kudos
Latest Reply
abhilash
New Contributor II
  • 2 kudos

Did you find any solution?

  • 2 kudos
Chris_Shehu
by Valued Contributor III
  • 5893 Views
  • 7 replies
  • 4 kudos

On-Behalf of tokens disabled for Azure Environments?

While trying to setup a Power BI connection to the Azure Delta Lake we ran into several issues around Service Principals. ​1) The API listed on the learn.microsoft site (link 1 below) indicates that there is an API you can use to create SP tokens. Wh...

  • 5893 Views
  • 7 replies
  • 4 kudos
Latest Reply
meetskorun
New Contributor II
  • 4 kudos

hello,i am new here from india, here to share some thoughts with you all

  • 4 kudos
6 More Replies
Simmy
by New Contributor II
  • 3163 Views
  • 3 replies
  • 1 kudos

Databricks Repos are pushing all changes to GitHub

When I make any changes within a repo, when I go to commit and push to GitHub if I uncheck any changes that I don't want pushed, still get pushed to Github. Any help would be appreciated

  • 3163 Views
  • 3 replies
  • 1 kudos
Latest Reply
Simmy
New Contributor II
  • 1 kudos

Hi @Vidula Khanna​ Problem is now resolved thanks, didn't have to do anything different, the functionality just started working as expected.

  • 1 kudos
2 More Replies
dcantos
by New Contributor II
  • 7527 Views
  • 5 replies
  • 0 kudos

IP Access list Databricks

Hello, good morning everyone, I have a problem, I have IP access List activated in my Databricks workspace and I have connections to powerBI and Azure DevOps, in Power BI I have already added all the public IP ranges that Azure gives me, which are ab...

  • 7527 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @David Cantos​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
4 More Replies
uv
by New Contributor II
  • 7068 Views
  • 3 replies
  • 2 kudos

Parquet to csv delta file

Hi Team, I have a parquet file in s3 bucket which is a delta file I am able to read it but I am unable to write it as a csv file.​getting the following error when i am trying to write:​A transaction log for Databricks Delta was found at `s3://path/a...

  • 7068 Views
  • 3 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @yuvesh kotiala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 2 kudos
2 More Replies
Saty
by New Contributor
  • 17133 Views
  • 3 replies
  • 1 kudos

Job is fails with java.lang.NoClassDefFoundError: Could not initialize class error

hi,It is scala code where we are connecting Redis to store (sparkcontext.toRedisKV) and i am also using scala udf . ihave excuted the same code in notebook without scala object and it works fine but everytime it fails when i am using same code in jar...

  • 17133 Views
  • 3 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Satish Kumbhar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 1 kudos
2 More Replies
Kyler
by New Contributor
  • 2020 Views
  • 2 replies
  • 0 kudos

ILT session

when is the next ILT session

  • 2020 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Edrian Kyle​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback ...

  • 0 kudos
1 More Replies
sribet
by New Contributor
  • 2112 Views
  • 2 replies
  • 0 kudos

Adding partition location to external table throwing exception

I have created external table like below.# create table spark.sql(f""" CREATE EXTERNAL TABLE IF NOT EXISTS {database_schema}.{tableName} USING PARQUET OPTIONS ( 'path' '{raw_storage}/{folder_path}', 'forward_spark_azur...

  • 2112 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @sri bet​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback will...

  • 0 kudos
1 More Replies
Dave_Nithio
by Contributor II
  • 3837 Views
  • 2 replies
  • 1 kudos

Skip Task Without Spinning Up Cluster

I have a Job Workflow with multiple sequential tasks executing R or Python scripts. Currently, we can skip one of these tasks (if it has already been run) by passing a parameter and skipping via the script. This requires a full spin up of a compute r...

  • 3837 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Dave Wilson​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...

  • 1 kudos
1 More Replies
PFA
by New Contributor
  • 7024 Views
  • 2 replies
  • 0 kudos

Why is the Import button greyed out when attempting to import a .ipynb file into Databricks?

The file import seems to work. The file name and size appear in the dialog box along with a green check mark. BUT only the cancel button is active. The import button is greyed out.Is there another step, action, or setting required to activate the i...

  • 7024 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Paul Auclair​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 0 kudos
1 More Replies
MaheshDR
by New Contributor II
  • 3483 Views
  • 2 replies
  • 0 kudos

Informatica Cloud mapping with Databricks connection failing with java.util.NoSuchElementException

Hi Team,When we tried to configure our source with Databricks table with Databricks connection on Informatica Cloud, we received below error.We already tried the suggestions mentioned in the below community post which seems to be similar error as our...

  • 3483 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Mahesh D​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!

  • 0 kudos
1 More Replies
674491
by New Contributor II
  • 2768 Views
  • 2 replies
  • 1 kudos

Is it possible to get a git access token on the databricks notebook.

I set my git access token with folloing the page.Get a Git access token & connect a remote repo to Databricks | Databricks on AWSI would like to get that git token on my databricks python notebook.Is it possible? Or, should I set same or another git ...

  • 2768 Views
  • 2 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @takeshi.kida​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...

  • 1 kudos
1 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels